Nov 03 22:06:39.990934 ip-10-0-139-254 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:06:40.355348 ip-10-0-139-254 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:06:40.355348 ip-10-0-139-254 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:06:40.355348 ip-10-0-139-254 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:06:40.355348 ip-10-0-139-254 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:06:40.359532 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359312 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:40.359532 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359533 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:40.359532 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359537 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359540 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359543 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359546 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359549 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359551 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359554 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359557 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359560 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359563 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359566 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359569 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359571 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359574 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359577 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359580 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359583 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359585 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359588 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359595 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:40.359626 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359599 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359601 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359604 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359606 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359609 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359612 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359614 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359617 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359619 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359622 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359625 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359627 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359632 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359635 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359638 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359653 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359656 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359659 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359661 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359664 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:40.360103 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359666 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359669 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359671 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359674 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359676 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359679 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359681 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359684 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359687 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359691 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359694 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359696 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359699 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359702 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359705 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359708 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359711 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359714 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359717 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:40.360579 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359719 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359722 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359724 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359727 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359730 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359733 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359735 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359738 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359740 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359743 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359745 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359751 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359755 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359758 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359761 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359764 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359766 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359769 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359772 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:40.361039 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359774 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359777 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359780 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359783 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359785 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359788 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359791 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359794 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359797 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359800 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359803 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359806 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359808 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359814 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359817 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359820 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359823 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359825 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359828 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:40.361604 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359831 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359833 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359836 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359838 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359841 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359843 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.359846 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360660 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360667 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360670 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360673 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360676 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360679 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360682 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360684 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360687 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360689 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360692 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360695 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360697 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:40.362072 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360700 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360703 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360706 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360708 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360711 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360714 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360716 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360719 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360722 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360724 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360726 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360729 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360732 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360734 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360737 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360739 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360742 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360744 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360747 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360750 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:40.362562 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360753 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360755 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360758 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360761 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360763 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360765 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360768 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360771 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360773 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360776 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360778 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360781 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360784 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360787 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360790 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360792 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360795 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360798 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360800 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:40.363076 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360803 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360805 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360809 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360812 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360814 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360817 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360819 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360822 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360824 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360827 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360829 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360832 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360834 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360837 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360840 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360842 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360845 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360847 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360851 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360854 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:40.363531 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360856 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360859 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360861 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360864 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360866 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360870 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360872 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360875 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360878 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360880 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360883 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360885 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360890 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360894 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360897 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360901 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360903 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360906 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360909 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:40.364085 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360912 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360914 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360917 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360920 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360922 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360925 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360927 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360930 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360933 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360936 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360939 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360941 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360944 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360950 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.360954 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361449 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361454 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361457 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361460 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:40.364558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361463 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361466 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361469 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361471 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361474 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361477 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361479 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361482 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361485 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361488 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361491 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361493 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361496 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361499 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361501 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361504 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361506 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361509 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361511 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361513 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:40.365043 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361520 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361523 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361525 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361529 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361533 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361536 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361539 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361541 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361544 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361546 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361549 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361552 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361554 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361557 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361559 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361562 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361564 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361567 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361570 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:40.365523 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361572 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361575 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361578 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361581 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361583 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361586 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361589 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361591 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361594 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361596 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361598 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361601 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361603 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361606 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361609 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361611 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361615 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361620 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361622 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:40.365998 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361625 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361628 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361630 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361633 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361636 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361652 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361655 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361658 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361660 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361663 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361666 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361668 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361671 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361674 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361677 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361679 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361681 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361684 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361687 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:40.366458 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361690 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361692 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361695 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361697 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361700 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361703 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361705 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361708 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361710 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361714 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361716 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361720 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361723 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361725 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361728 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361730 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361733 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361735 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361738 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361743 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:40.366930 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361747 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361749 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361752 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361755 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.361757 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361855 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361863 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361870 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361875 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361879 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361882 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361887 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361891 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361894 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361897 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361901 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361904 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361907 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361910 2581 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361913 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361916 2581 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361919 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361923 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:06:40.367410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361928 2581 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361931 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361934 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361937 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361960 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361966 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361969 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361973 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361977 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361980 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361984 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361987 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361991 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361993 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.361998 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362001 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362004 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362007 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362010 2581 flags.go:64] FLAG: --enable-server="true" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362013 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362018 2581 flags.go:64] FLAG: --event-burst="100" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362022 2581 flags.go:64] FLAG: --event-qps="50" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362025 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362028 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:06:40.367958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362031 2581 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362035 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362039 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362042 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362045 2581 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362048 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362051 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362054 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362059 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362062 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362065 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362068 2581 flags.go:64] FLAG: --feature-gates="" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362073 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362076 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362080 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362084 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362087 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362090 2581 flags.go:64] FLAG: --help="false" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362093 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362097 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362100 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362103 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362106 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:06:40.368630 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362109 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362112 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362115 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362119 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362122 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362124 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362128 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362130 2581 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362134 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362137 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362140 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362142 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362146 2581 flags.go:64] FLAG: --lock-file="" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362148 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362151 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362155 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362161 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362165 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362168 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362172 2581 flags.go:64] FLAG: --logging-format="text" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362175 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362178 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362181 2581 flags.go:64] FLAG: --manifest-url="" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362184 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362189 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:06:40.369201 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362192 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362196 2581 flags.go:64] FLAG: --max-pods="110" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362198 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362201 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362204 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362207 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362210 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362213 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362216 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362224 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362227 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362230 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362233 2581 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362236 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362239 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362242 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362245 2581 flags.go:64] FLAG: --port="10250" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362248 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362251 2581 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-047375d5d7ca8f156" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362254 2581 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362257 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362260 2581 flags.go:64] FLAG: --register-node="true" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362263 2581 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362267 2581 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362271 2581 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362274 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:06:40.369816 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362278 2581 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362282 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362285 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362288 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362291 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362295 2581 flags.go:64] FLAG: --runonce="false" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362298 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362301 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362304 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362308 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362310 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362314 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362317 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362320 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362323 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362326 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362329 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362332 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362335 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362338 2581 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362341 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362346 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362349 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362352 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362356 2581 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:06:40.370415 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362359 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362362 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362365 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362368 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362371 2581 flags.go:64] FLAG: --v="2" Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362377 2581 flags.go:64] FLAG: --version="false" Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362381 2581 flags.go:64] FLAG: --vmodule="" Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362385 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362389 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362527 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362531 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362534 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362538 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362541 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362544 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362546 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362549 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362552 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362554 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362557 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362559 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362563 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:40.371030 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362565 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362568 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362571 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362574 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362576 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362579 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362582 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362584 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362587 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362590 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362593 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362597 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362600 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362604 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362607 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362610 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362613 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362616 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362618 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:40.371558 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362622 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362624 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362627 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362631 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362634 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362636 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362651 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362654 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362657 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362660 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362662 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362665 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362668 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362670 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362673 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362676 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362679 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362682 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362684 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362687 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:40.372024 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362689 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362692 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362695 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362697 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362700 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362702 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362705 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362707 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362711 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362714 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362717 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362720 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362722 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362726 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362729 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362733 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362736 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362738 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362741 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362744 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:40.372498 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362746 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362749 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362751 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362754 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362756 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362759 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362762 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362764 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362767 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362769 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362772 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362774 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362777 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362779 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362782 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362785 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362787 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362790 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362792 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362795 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:40.373016 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362797 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362800 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362802 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362806 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362810 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362814 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362817 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362821 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362824 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362826 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362830 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362832 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362835 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362838 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.362865 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:06:40.373493 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362912 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362917 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362920 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362922 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362925 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362927 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362930 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362933 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362935 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362938 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362941 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362943 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362947 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362949 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362952 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362955 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362957 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362960 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362962 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:40.373867 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362965 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362967 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362970 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362972 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362975 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362980 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362985 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362988 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362991 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362994 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362996 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.362999 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363001 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363004 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363006 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363009 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363013 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363016 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363018 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:40.374306 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363021 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363025 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363028 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363030 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363033 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363035 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363038 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363040 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363043 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363046 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363049 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363051 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363054 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363056 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363059 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363061 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363064 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363066 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363069 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363073 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:40.374824 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363076 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363079 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363082 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363084 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363087 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363090 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363093 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363095 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363098 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363101 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363104 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363107 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363110 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363112 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363114 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363117 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363120 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363123 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363125 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363128 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:40.375310 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363130 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363133 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363135 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363138 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363140 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363143 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363146 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363148 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363151 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363153 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363156 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363160 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363163 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363166 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363168 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363171 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363173 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363178 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363181 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:40.375806 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363184 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363186 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363190 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363193 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363195 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363198 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363202 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363205 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.363207 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.370675 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.370692 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.371390 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.371397 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371481 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371488 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371491 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371494 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371496 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371499 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:40.376246 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371502 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371505 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371508 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371511 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371513 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371517 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371519 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371529 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371531 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371536 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371540 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371543 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371547 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371549 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371552 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371555 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371558 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371561 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371564 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:40.376781 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371567 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371571 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371575 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371579 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371582 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371584 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371587 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371589 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371592 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371595 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371597 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371600 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371602 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371605 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371607 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371610 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371612 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371615 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371618 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:40.377227 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371620 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371623 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371652 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371656 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371658 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371661 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371664 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371666 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371669 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371672 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371674 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371677 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371680 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371683 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371685 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371688 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371691 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371693 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371696 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371698 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:40.377705 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371701 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371703 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371706 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371712 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371715 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371717 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371720 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371722 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371725 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371727 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371730 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371734 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371736 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371739 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371742 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371750 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371753 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371756 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371759 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371761 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:40.378160 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371763 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371766 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371768 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371771 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371773 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371776 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371779 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371781 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371784 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371786 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371789 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371791 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371794 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371796 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371799 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371802 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371804 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371807 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371809 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:40.378670 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371812 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371814 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.371816 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.371850 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372054 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372060 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372063 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372066 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372068 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372077 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372080 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372082 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372085 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372088 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372092 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:40.379154 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372095 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372098 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372101 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372104 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372107 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372109 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372112 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372114 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372117 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372120 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372122 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372125 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372127 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372130 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372132 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372135 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372137 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372140 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372142 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372144 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:40.379506 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372147 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372150 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372153 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372155 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372158 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372161 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372163 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372171 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372174 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372177 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372180 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372184 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372188 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372190 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372193 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372196 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372198 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372201 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372203 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:40.379995 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372206 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372208 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372211 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372213 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372216 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372218 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372221 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372223 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372226 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372228 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372231 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372233 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372236 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372238 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372243 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372246 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372249 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372251 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372254 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372256 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:40.380454 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372259 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372267 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372270 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372272 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372275 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372277 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372280 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372282 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372285 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372288 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372290 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372293 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372295 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372298 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372301 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372303 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372306 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372309 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372311 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372314 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:40.381118 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372316 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372319 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372322 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372324 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372327 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372329 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372334 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372336 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372339 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372342 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372345 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372348 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372350 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372353 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372361 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372364 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:40.382129 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.372387 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372438 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372442 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372446 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372448 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372451 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372453 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372456 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372458 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372461 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372464 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372466 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372469 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372471 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372474 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372476 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372479 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372481 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372484 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372487 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:40.382609 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372489 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372491 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372494 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372499 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372502 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372505 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372507 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372510 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372513 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372516 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372519 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372527 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372530 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372533 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372535 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372538 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372540 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372543 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372545 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:40.383179 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372548 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372550 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372553 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372555 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372558 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372562 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372565 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372568 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372571 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372575 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372577 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372580 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372582 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372585 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372588 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372591 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372595 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372597 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372606 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:40.383667 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372609 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372611 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372614 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372616 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372619 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372622 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372630 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372633 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372635 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372638 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372654 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372657 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372661 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372664 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372666 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372669 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372671 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372675 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372679 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:40.384138 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372682 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372686 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372688 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372691 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372693 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372696 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372699 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372701 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372704 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372707 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372710 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372713 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372715 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372718 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372721 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372723 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372726 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372728 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372731 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:40.384612 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372733 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372742 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372745 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372747 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372750 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372752 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372756 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372759 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372762 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372764 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:40.372768 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.372831 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.377085 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.377920 2581 server.go:1014] "Starting client certificate rotation" Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.378029 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.378073 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:06:40.385099 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.382272 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:06:40.387519 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.387494 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:06:40.398347 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.398320 2581 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:06:40.401818 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.401801 2581 log.go:25] "Validated CRI v1 image API" Nov 03 22:06:40.402705 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.402678 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:06:40.406625 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.406605 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ca621f77-771f-4b8c-8a2b-690734981466:/dev/nvme0n1p4 f8a8aa0f-c912-477a-a7a2-18f39b7c48b1:/dev/nvme0n1p3] Nov 03 22:06:40.406688 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.406625 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 22:06:40.412658 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.412533 2581 manager.go:217] Machine: {Timestamp:2026-11-03 22:06:40.411183183 +0000 UTC m=+0.309808112 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101019 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa93931d0687f87b29780675c8062 SystemUUID:ec2aa939-31d0-687f-87b2-9780675c8062 BootID:ba326697-dcb3-46f8-ac96-02c76373c5e7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d5:ed:97:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d5:ed:97:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:de:50:99:9b:20 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:06:40.412736 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.412660 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:06:40.412797 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.412785 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:06:40.415049 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.415017 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:06:40.415232 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.415050 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-254.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:06:40.415312 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.415237 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:06:40.415312 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.415248 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:06:40.415312 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.415301 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:06:40.415418 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.415327 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:06:40.416351 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.416339 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:06:40.420347 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.420331 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:06:40.420489 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.420477 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:06:40.421297 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.421286 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:06:40.421351 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.421306 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:06:40.421351 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.421327 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:06:40.421351 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.421341 2581 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:06:40.421474 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.421354 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:06:40.421772 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.421756 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:06:40.423895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.423877 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:06:40.429333 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.429319 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:06:40.430141 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430128 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:06:40.430184 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430159 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:06:40.430327 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430309 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:06:40.430327 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430326 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:06:40.430327 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430334 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:06:40.430493 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430340 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:06:40.430493 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430346 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:06:40.430493 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430352 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:06:40.430902 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430891 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:06:40.430934 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430902 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:06:40.430934 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430911 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:06:40.430934 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430917 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:06:40.430934 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430932 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:06:40.431112 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.430941 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:06:40.431535 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.431525 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:06:40.431566 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.431538 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:06:40.435314 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.435301 2581 server.go:1263] "Started kubelet" Nov 03 22:06:40.435554 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.435500 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:06:40.435554 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.435454 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:06:40.435722 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.435706 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:06:40.439048 ip-10-0-139-254 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:06:40.442270 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.441905 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:06:40.444045 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.443985 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:06:40.445829 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.445810 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 22:06:40.447942 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.447928 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:06:40.448422 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.448405 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:06:40.448605 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.448583 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:06:40.449141 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449121 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:06:40.449141 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449139 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:06:40.449285 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449121 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:06:40.449285 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449242 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:06:40.449285 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449257 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:06:40.449285 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449266 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:06:40.449285 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449258 2581 factory.go:55] Registering systemd factory Nov 03 22:06:40.449285 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449285 2581 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:06:40.449596 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449582 2581 factory.go:153] Registering CRI-O factory Nov 03 22:06:40.449596 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449594 2581 factory.go:223] Registration of the crio container factory successfully Nov 03 22:06:40.449729 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449608 2581 factory.go:103] Registering Raw factory Nov 03 22:06:40.449729 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.449618 2581 manager.go:1199] Started watching for new ooms in manager Nov 03 22:06:40.450505 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.450489 2581 manager.go:319] Starting recovery of all containers Nov 03 22:06:40.461459 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.461439 2581 manager.go:324] Recovery completed Nov 03 22:06:40.467131 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.467112 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:40.469297 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.469274 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:40.469377 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.469308 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:40.469377 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.469324 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:40.470002 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.469984 2581 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:06:40.470090 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.470007 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:06:40.470090 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.469990 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:40.470183 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.470025 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:06:40.470845 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.470825 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:40.470924 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.470858 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:40.470924 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.470872 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:40.472259 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.472243 2581 policy_none.go:50] "Start" Nov 03 22:06:40.472341 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.472263 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:06:40.472341 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.472276 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:06:40.474269 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.474256 2581 policy_none.go:44] "Start" Nov 03 22:06:40.509080 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.509053 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:06:40.511758 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.511744 2581 manager.go:341] "Starting Device Plugin manager" Nov 03 22:06:40.511825 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.511775 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:06:40.511825 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.511785 2581 server.go:87] "Starting device plugin registration server" Nov 03 22:06:40.512042 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.512030 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:06:40.512088 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.512042 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:06:40.512168 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.512151 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:06:40.512250 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.512233 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:06:40.512250 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.512246 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:06:40.512877 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.512854 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:06:40.512938 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.512928 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:06:40.568253 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.568218 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-254.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 22:06:40.568253 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.568226 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-254.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 22:06:40.570384 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.568211 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-254.us-west-2.compute.internal.18e4a84d0a1317d2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-254.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-254.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:06:40.435271634 +0000 UTC m=+0.333896561,LastTimestamp:2026-11-03 22:06:40.435271634 +0000 UTC m=+0.333896561,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-254.us-west-2.compute.internal,}" Nov 03 22:06:40.581093 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.581058 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:06:40.582284 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.582270 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:06:40.582353 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.582294 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:06:40.582353 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.582315 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:06:40.582451 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.582355 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 22:06:40.612359 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.612230 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:40.613242 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.613218 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:40.613353 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.613258 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:40.613353 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.613279 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:40.613353 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.613311 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.638113 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.638082 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 22:06:40.638247 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.638159 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 22:06:40.638417 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.638342 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-254.us-west-2.compute.internal.18e4a84d0c1a42c2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-254.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-254.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-254.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:06:40.46929581 +0000 UTC m=+0.367920738,LastTimestamp:2026-11-03 22:06:40.46929581 +0000 UTC m=+0.367920738,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-254.us-west-2.compute.internal,}" Nov 03 22:06:40.683109 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.683075 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal"] Nov 03 22:06:40.683285 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.683147 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:40.684271 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.684252 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:40.684326 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.684282 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:40.684326 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.684293 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:40.686622 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.686610 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:40.686822 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.686805 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.686901 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.686846 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:40.687708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.687690 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:40.687708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.687702 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:40.687828 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.687722 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:40.687828 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.687733 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:40.687828 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.687722 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:40.687828 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.687802 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:40.690039 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.690023 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.690102 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.690046 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:40.690102 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.690083 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:06:40.690198 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.690141 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nzg2h" Nov 03 22:06:40.692801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.692785 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:40.692879 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.692811 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:40.692879 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.692821 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:40.708328 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.708303 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.711844 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.711824 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.725961 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.725863 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-254.us-west-2.compute.internal.18e4a84d0c1a8f0b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-254.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-254.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-254.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:06:40.469315339 +0000 UTC m=+0.367940270,LastTimestamp:2026-11-03 22:06:40.469315339 +0000 UTC m=+0.367940270,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-254.us-west-2.compute.internal,}" Nov 03 22:06:40.726059 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.725933 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-254.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 22:06:40.726097 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.726059 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 03 22:06:40.752089 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.752048 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.752089 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.752090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.752271 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.752111 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/815f1999fc2bcba4fa27ffca21498e4f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"815f1999fc2bcba4fa27ffca21498e4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.784287 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.784259 2581 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-254.us-west-2.compute.internal\" not found" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.853220 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.853184 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.853220 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.853219 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.853435 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.853238 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/815f1999fc2bcba4fa27ffca21498e4f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"815f1999fc2bcba4fa27ffca21498e4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.853435 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.853292 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.853435 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.853305 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.853435 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.853333 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/815f1999fc2bcba4fa27ffca21498e4f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"815f1999fc2bcba4fa27ffca21498e4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.865672 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:40.865589 2581 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-254.us-west-2.compute.internal\" not found" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:40.984386 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.984350 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:40.985561 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.985543 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:40.985668 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.985575 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:40.985668 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.985591 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:40.985668 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:40.985622 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:41.012146 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.012113 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:41.015696 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.015673 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:41.036822 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.036799 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:41.036935 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:41.036827 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-254.us-west-2.compute.internal\": node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:06:41.156528 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:41.156457 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:06:41.256633 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:41.256604 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:06:41.357420 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:41.357390 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:06:41.378545 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.378495 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 22:06:41.378814 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:41.378725 2581 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3890ba9e34074c9bb3a6df7076aca19-87013095a60c6b5c.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-254.us-west-2.compute.internal.18e4a84d0c1a8f0b\": read tcp 10.0.139.254:45946->16.58.171.225:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-254.us-west-2.compute.internal.18e4a84d0c1a8f0b default 5114 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-254.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-254.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-254.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:06:40 +0000 UTC,LastTimestamp:2026-11-03 22:06:40.684287425 +0000 UTC m=+0.582912352,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-254.us-west-2.compute.internal,}" Nov 03 22:06:41.458077 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:41.457997 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:06:41.559177 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:41.559137 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:06:41.659603 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:41.659562 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:06:41.695543 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.695499 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:01:40 +0000 UTC" deadline="2028-06-15 19:08:44.869608338 +0000 UTC" Nov 03 22:06:41.695543 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.695532 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14157h2m3.174079987s" Nov 03 22:06:41.715167 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.715104 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:41.716249 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.716228 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:41.716345 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.716270 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:41.716345 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.716288 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:41.760000 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:41.759967 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:06:41.793165 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.793136 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:41.829178 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.829147 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:41.849801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.849763 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:41.914337 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.914304 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:06:41.915980 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.915965 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:06:41.976188 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:41.976108 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:06:42.032338 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.032307 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:42.241461 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:42.241279 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815f1999fc2bcba4fa27ffca21498e4f.slice/crio-85f69c9fc91998bfb5d5cdf395dfedfd5d780b67180868f2909ad106762380be WatchSource:0}: Error finding container 85f69c9fc91998bfb5d5cdf395dfedfd5d780b67180868f2909ad106762380be: Status 404 returned error can't find the container with id 85f69c9fc91998bfb5d5cdf395dfedfd5d780b67180868f2909ad106762380be Nov 03 22:06:42.241700 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:42.241661 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00032b9e4863070a848e5aad12046474.slice/crio-8ce713afc7eaf63a516c415eff23041ed2464eefebb24c6ca371432da39f7410 WatchSource:0}: Error finding container 8ce713afc7eaf63a516c415eff23041ed2464eefebb24c6ca371432da39f7410: Status 404 returned error can't find the container with id 8ce713afc7eaf63a516c415eff23041ed2464eefebb24c6ca371432da39f7410 Nov 03 22:06:42.247029 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.247008 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:06:42.285182 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.285158 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:42.422760 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.422717 2581 apiserver.go:52] "Watching apiserver" Nov 03 22:06:42.450118 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.450091 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 22:06:42.537525 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.537445 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:06:42.537797 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.537775 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-b94tl","kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-h5mwd","openshift-multus/multus-r2k9j","openshift-dns/node-resolver-6q9pz","openshift-network-diagnostics/network-check-target-bljnw","openshift-ovn-kubernetes/ovnkube-node-5gw2b","kube-system/global-pull-secret-syncer-7zmnf","openshift-image-registry/node-ca-k98wl","openshift-multus/network-metrics-daemon-4m4qq","openshift-network-operator/iptables-alerter-lr6gj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-f8hqj"] Nov 03 22:06:42.542066 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.542047 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.544335 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.544314 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.546639 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.546623 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.548994 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.548977 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:06:42.553475 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.553452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:42.553589 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:42.553557 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:06:42.556108 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.555985 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:06:42.558426 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.558403 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:06:42.560787 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.560767 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.563142 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.563123 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:42.563231 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:42.563209 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:06:42.565148 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-env-overrides\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.565235 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-systemd\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.565235 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565172 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-conf-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.565235 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-system-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.565235 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-etc-kubernetes\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.565439 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgzdr\" (UniqueName: \"kubernetes.io/projected/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-kube-api-access-xgzdr\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:06:42.565439 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565330 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-node-log\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.565511 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565466 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovn-node-metrics-cert\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.565511 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565491 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:06:42.565624 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565493 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-multus\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.565686 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-kubelet\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.565762 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565702 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-socket-dir-parent\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.565762 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565729 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-script-lib\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.565762 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565757 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-k8s-cni-cncf-io\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.565920 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565853 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-netns\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.565920 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565888 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-daemon-config\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.565920 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-tmp-dir\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:06:42.566157 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565942 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-lib-modules\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.566157 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565965 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-hostroot\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.566157 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.565989 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.566157 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566013 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-config\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.566157 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566037 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysconfig\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.566157 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-sys\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.566157 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-cnibin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.566157 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566110 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-var-lib-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.566157 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.566157 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566152 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-hosts-file\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566169 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-netd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566184 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-host\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566199 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-etc-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566243 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-ovn\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566270 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566299 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-tuned\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566327 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-cni-binary-copy\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566355 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-bin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566388 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-netns\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-log-socket\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566421 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-multus-certs\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566439 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-systemd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566461 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-bin\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566486 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-modprobe-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566509 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-var-lib-kubelet\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566576 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-kubernetes\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.566609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566612 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-systemd-units\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.567154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-slash\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.567154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566679 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vxk\" (UniqueName: \"kubernetes.io/projected/153d28e6-5d08-4064-9c61-2dd05100b2d0-kube-api-access-44vxk\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.567154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566718 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-tmp\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.567154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566749 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvn9\" (UniqueName: \"kubernetes.io/projected/6b091fb4-183e-41e9-a408-84fd1ca8f184-kube-api-access-jqvn9\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.567154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566788 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.567154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566820 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-conf\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.567154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-kubelet\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.567154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566884 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-run\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.567154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fsqb\" (UniqueName: \"kubernetes.io/projected/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-kube-api-access-2fsqb\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.567154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566943 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwcfj\" (UniqueName: \"kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj\") pod \"network-check-target-bljnw\" (UID: \"6b06086c-7464-4be0-a0c6-7568cb1dcd30\") " pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:42.567154 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.566972 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.567888 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.567875 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.570352 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.570333 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:42.570423 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:42.570392 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:06:42.587179 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.587135 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" event={"ID":"815f1999fc2bcba4fa27ffca21498e4f","Type":"ContainerStarted","Data":"85f69c9fc91998bfb5d5cdf395dfedfd5d780b67180868f2909ad106762380be"} Nov 03 22:06:42.588399 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.588372 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" event={"ID":"00032b9e4863070a848e5aad12046474","Type":"ContainerStarted","Data":"8ce713afc7eaf63a516c415eff23041ed2464eefebb24c6ca371432da39f7410"} Nov 03 22:06:42.599402 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.599374 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:06:42.599402 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.599394 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:06:42.599636 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.599423 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:06:42.599636 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.599430 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:06:42.599636 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.599466 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z5g9w\"" Nov 03 22:06:42.599636 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.599634 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:06:42.599971 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.599950 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:06:42.600449 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.600422 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:06:42.600925 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.600898 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:06:42.601010 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.600947 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dgskz\"" Nov 03 22:06:42.604243 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.603126 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:06:42.604243 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.603160 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bkplg\"" Nov 03 22:06:42.604243 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.603265 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:06:42.604243 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.603326 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:06:42.604243 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.603401 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:06:42.605322 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.605301 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ckrtj\"" Nov 03 22:06:42.605408 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.605369 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:06:42.605752 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.605733 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:06:42.612470 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.612453 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:06:42.612470 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.612452 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:06:42.612599 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.612522 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:06:42.612599 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.612526 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8pftz\"" Nov 03 22:06:42.614549 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.614534 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m27qh\"" Nov 03 22:06:42.648481 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.648457 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:06:42.650237 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.650219 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:06:42.653394 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653373 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:06:42.653543 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653527 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5mdhj\"" Nov 03 22:06:42.653611 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653597 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:06:42.653683 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653650 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:06:42.653756 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653685 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:06:42.653756 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653702 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dngqk\"" Nov 03 22:06:42.653756 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653710 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:06:42.653756 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653715 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:06:42.653756 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653702 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:06:42.653947 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653814 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j54w4\"" Nov 03 22:06:42.653947 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653816 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:06:42.654051 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.653975 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:06:42.654051 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.654020 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7v77v" Nov 03 22:06:42.654051 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.654028 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:06:42.667284 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667260 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76b201e3-7c0b-4810-9baa-f6e054cf0826-host\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:06:42.667344 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667291 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cnibin\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.667344 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667309 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzcb8\" (UniqueName: \"kubernetes.io/projected/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-kube-api-access-dzcb8\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.667344 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-registration-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.667431 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667378 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-systemd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.667464 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667444 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-systemd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.667500 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667482 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-bin\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.667534 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667502 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-modprobe-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.667569 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667533 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-var-lib-kubelet\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.667569 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667550 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-bin\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.667666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-host-slash\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:06:42.667666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-kubernetes\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.667666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-systemd-units\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.667666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667671 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-var-lib-kubelet\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.667895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667659 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-slash\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.667895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667710 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-slash\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.667895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667713 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-kubernetes\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.667895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667703 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-modprobe-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.667895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vxk\" (UniqueName: \"kubernetes.io/projected/153d28e6-5d08-4064-9c61-2dd05100b2d0-kube-api-access-44vxk\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.667895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667750 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-systemd-units\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.667895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-tmp\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.667895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667796 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvn9\" (UniqueName: \"kubernetes.io/projected/6b091fb4-183e-41e9-a408-84fd1ca8f184-kube-api-access-jqvn9\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.667895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667824 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-binary-copy\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.667895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667850 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.667895 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667883 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-conf\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667908 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-kubelet\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667933 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-run\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fsqb\" (UniqueName: \"kubernetes.io/projected/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-kube-api-access-2fsqb\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667964 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667987 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcfj\" (UniqueName: \"kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj\") pod \"network-check-target-bljnw\" (UID: \"6b06086c-7464-4be0-a0c6-7568cb1dcd30\") " pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.667992 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-kubelet\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668016 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9t4x\" (UniqueName: \"kubernetes.io/projected/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-kube-api-access-f9t4x\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668045 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668058 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-conf\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-iptables-alerter-script\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668101 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-kubelet-config\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668116 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-env-overrides\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668118 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668155 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-systemd\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668183 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-conf-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.668407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668187 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-run\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-system-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668244 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-etc-kubernetes\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668265 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-system-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668272 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgzdr\" (UniqueName: \"kubernetes.io/projected/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-kube-api-access-xgzdr\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668272 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-conf-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668281 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-systemd\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668292 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc56646d-7901-4a14-a4af-46d2582ccc5c-konnectivity-ca\") pod \"konnectivity-agent-b94tl\" (UID: \"cc56646d-7901-4a14-a4af-46d2582ccc5c\") " pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668310 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-etc-kubernetes\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668316 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-node-log\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668341 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovn-node-metrics-cert\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-multus\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668374 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-kubelet\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668374 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-node-log\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668400 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsncz\" (UniqueName: \"kubernetes.io/projected/76b201e3-7c0b-4810-9baa-f6e054cf0826-kube-api-access-nsncz\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668432 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94d26\" (UniqueName: \"kubernetes.io/projected/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-kube-api-access-94d26\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668439 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-multus\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.669206 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668452 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-kubelet\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668458 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-socket-dir-parent\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-script-lib\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-k8s-cni-cncf-io\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668594 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-socket-dir-parent\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668645 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-env-overrides\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-netns\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668697 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-k8s-cni-cncf-io\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668760 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-netns\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-daemon-config\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668805 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668823 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668854 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-socket-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668901 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-dbus\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668944 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-tmp-dir\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-lib-modules\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.670016 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.668992 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-hostroot\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669015 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc56646d-7901-4a14-a4af-46d2582ccc5c-agent-certs\") pod \"konnectivity-agent-b94tl\" (UID: \"cc56646d-7901-4a14-a4af-46d2582ccc5c\") " pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669037 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-config\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669084 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-script-lib\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669092 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-lib-modules\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysconfig\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669124 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-sys\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669155 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-cnibin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669178 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysconfig\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669180 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-system-cni-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669211 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfvn\" (UniqueName: \"kubernetes.io/projected/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kube-api-access-kwfvn\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-var-lib-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669251 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-cnibin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669264 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669303 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/76b201e3-7c0b-4810-9baa-f6e054cf0826-serviceca\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:06:42.670666 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669345 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-hostroot\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669348 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-tmp-dir\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669361 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-hosts-file\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669383 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669436 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-var-lib-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669451 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-daemon-config\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669467 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669498 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-sys\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669600 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-hosts-file\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-netd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669646 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-host\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669683 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-netd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-device-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669720 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-sys-fs\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669725 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-host\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669782 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-etc-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669810 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-etc-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671219 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669815 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-ovn\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669820 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-config\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669856 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-ovn\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669876 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-tuned\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669892 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669902 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-cni-binary-copy\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-bin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669974 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-bin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.669979 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.670047 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-netns\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.670074 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-log-socket\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.670098 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-multus-certs\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.670145 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-netns\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.670168 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-multus-certs\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.670188 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-log-socket\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.671801 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.670349 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-cni-binary-copy\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.672307 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.672102 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovn-node-metrics-cert\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.672307 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.672194 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-tmp\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.672307 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.672229 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-tuned\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.727640 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:42.727614 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:42.727640 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:42.727638 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:42.727640 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:42.727665 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xwcfj for pod openshift-network-diagnostics/network-check-target-bljnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:42.727882 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:42.727745 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj podName:6b06086c-7464-4be0-a0c6-7568cb1dcd30 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:43.227723848 +0000 UTC m=+3.126348785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xwcfj" (UniqueName: "kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj") pod "network-check-target-bljnw" (UID: "6b06086c-7464-4be0-a0c6-7568cb1dcd30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:42.729889 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.729857 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgzdr\" (UniqueName: \"kubernetes.io/projected/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-kube-api-access-xgzdr\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:06:42.729889 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.729871 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vxk\" (UniqueName: \"kubernetes.io/projected/153d28e6-5d08-4064-9c61-2dd05100b2d0-kube-api-access-44vxk\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.730022 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.729868 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvn9\" (UniqueName: \"kubernetes.io/projected/6b091fb4-183e-41e9-a408-84fd1ca8f184-kube-api-access-jqvn9\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.730232 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.730215 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fsqb\" (UniqueName: \"kubernetes.io/projected/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-kube-api-access-2fsqb\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.771379 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771351 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-device-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.771488 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771381 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-sys-fs\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.771488 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.771488 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.771488 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771482 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771485 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-sys-fs\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771487 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-device-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771513 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76b201e3-7c0b-4810-9baa-f6e054cf0826-host\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cnibin\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcb8\" (UniqueName: \"kubernetes.io/projected/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-kube-api-access-dzcb8\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771583 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76b201e3-7c0b-4810-9baa-f6e054cf0826-host\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771587 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771599 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cnibin\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-registration-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-host-slash\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-binary-copy\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.771708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771660 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-registration-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771721 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9t4x\" (UniqueName: \"kubernetes.io/projected/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-kube-api-access-f9t4x\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771746 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-host-slash\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771801 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-iptables-alerter-script\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771817 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-kubelet-config\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771835 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc56646d-7901-4a14-a4af-46d2582ccc5c-konnectivity-ca\") pod \"konnectivity-agent-b94tl\" (UID: \"cc56646d-7901-4a14-a4af-46d2582ccc5c\") " pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771851 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsncz\" (UniqueName: \"kubernetes.io/projected/76b201e3-7c0b-4810-9baa-f6e054cf0826-kube-api-access-nsncz\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771866 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94d26\" (UniqueName: \"kubernetes.io/projected/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-kube-api-access-94d26\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771882 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771903 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.771925 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772044 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-socket-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772075 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-dbus\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772102 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc56646d-7901-4a14-a4af-46d2582ccc5c-agent-certs\") pod \"konnectivity-agent-b94tl\" (UID: \"cc56646d-7901-4a14-a4af-46d2582ccc5c\") " pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772045 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-kubelet-config\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:42.772111 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772131 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.772280 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772161 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-dbus\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772164 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-system-cni-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:42.772207 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret podName:a0475429-6780-4dd9-8185-199a83ce41f5 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:43.272161571 +0000 UTC m=+3.170786502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret") pod "global-pull-secret-syncer-7zmnf" (UID: "a0475429-6780-4dd9-8185-199a83ce41f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772234 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-system-cni-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfvn\" (UniqueName: \"kubernetes.io/projected/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kube-api-access-kwfvn\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772284 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/76b201e3-7c0b-4810-9baa-f6e054cf0826-serviceca\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772289 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-socket-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772311 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772323 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-binary-copy\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772409 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:42.772416 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:42.772474 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs podName:77d5bb8c-19f0-4d74-b146-ae9d69aebcdf nodeName:}" failed. No retries permitted until 2026-11-03 22:06:43.272458199 +0000 UTC m=+3.171083113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs") pod "network-metrics-daemon-4m4qq" (UID: "77d5bb8c-19f0-4d74-b146-ae9d69aebcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772453 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-iptables-alerter-script\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772493 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772512 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc56646d-7901-4a14-a4af-46d2582ccc5c-konnectivity-ca\") pod \"konnectivity-agent-b94tl\" (UID: \"cc56646d-7901-4a14-a4af-46d2582ccc5c\") " pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772792 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/76b201e3-7c0b-4810-9baa-f6e054cf0826-serviceca\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:06:42.772873 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.772847 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.774307 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.774291 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc56646d-7901-4a14-a4af-46d2582ccc5c-agent-certs\") pod \"konnectivity-agent-b94tl\" (UID: \"cc56646d-7901-4a14-a4af-46d2582ccc5c\") " pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:06:42.829728 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.829657 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsncz\" (UniqueName: \"kubernetes.io/projected/76b201e3-7c0b-4810-9baa-f6e054cf0826-kube-api-access-nsncz\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:06:42.829886 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.829793 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfvn\" (UniqueName: \"kubernetes.io/projected/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kube-api-access-kwfvn\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.829886 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.829806 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94d26\" (UniqueName: \"kubernetes.io/projected/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-kube-api-access-94d26\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:42.830001 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.829908 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzcb8\" (UniqueName: \"kubernetes.io/projected/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-kube-api-access-dzcb8\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.830001 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.829980 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9t4x\" (UniqueName: \"kubernetes.io/projected/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-kube-api-access-f9t4x\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:06:42.866842 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.866812 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:06:42.869960 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.869940 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r2k9j" Nov 03 22:06:42.877109 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:42.877083 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8a5faf_6b8f_4340_8b4c_945f8d67e008.slice/crio-ed72365a778a55a8b992ac6c62788f88a0cc69ca43e0bdcc7e3e3bc37095e5dd WatchSource:0}: Error finding container ed72365a778a55a8b992ac6c62788f88a0cc69ca43e0bdcc7e3e3bc37095e5dd: Status 404 returned error can't find the container with id ed72365a778a55a8b992ac6c62788f88a0cc69ca43e0bdcc7e3e3bc37095e5dd Nov 03 22:06:42.877345 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.877310 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:06:42.878128 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.878109 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:06:42.880404 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:42.880381 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b091fb4_183e_41e9_a408_84fd1ca8f184.slice/crio-94005349a5bfe1a9a5137505223eefe7d3ea3d7ccec33d0cb5c8dddab9eaf14e WatchSource:0}: Error finding container 94005349a5bfe1a9a5137505223eefe7d3ea3d7ccec33d0cb5c8dddab9eaf14e: Status 404 returned error can't find the container with id 94005349a5bfe1a9a5137505223eefe7d3ea3d7ccec33d0cb5c8dddab9eaf14e Nov 03 22:06:42.880533 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.880401 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:06:42.886494 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.886472 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:06:42.888261 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:42.888242 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153d28e6_5d08_4064_9c61_2dd05100b2d0.slice/crio-550f19f0fa67d96d43c1ab43cd2db377b82f2878eda402cf7a5d968cc7fcf04b WatchSource:0}: Error finding container 550f19f0fa67d96d43c1ab43cd2db377b82f2878eda402cf7a5d968cc7fcf04b: Status 404 returned error can't find the container with id 550f19f0fa67d96d43c1ab43cd2db377b82f2878eda402cf7a5d968cc7fcf04b Nov 03 22:06:42.890825 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:42.890777 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd381f494_4ddc_4d70_9d0f_f27d6b3a7d8d.slice/crio-2603ad2f8efbb8703e80500f47b3dc486eb54bff35395fcdfa0c9883009e8286 WatchSource:0}: Error finding container 2603ad2f8efbb8703e80500f47b3dc486eb54bff35395fcdfa0c9883009e8286: Status 404 returned error can't find the container with id 2603ad2f8efbb8703e80500f47b3dc486eb54bff35395fcdfa0c9883009e8286 Nov 03 22:06:42.894549 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:42.894528 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b201e3_7c0b_4810_9baa_f6e054cf0826.slice/crio-c6094815b5ac2e90de4c5282bb5ccd036fb0758d0919f6ee44149a20a5f7bd31 WatchSource:0}: Error finding container c6094815b5ac2e90de4c5282bb5ccd036fb0758d0919f6ee44149a20a5f7bd31: Status 404 returned error can't find the container with id c6094815b5ac2e90de4c5282bb5ccd036fb0758d0919f6ee44149a20a5f7bd31 Nov 03 22:06:42.895369 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:42.895350 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc56646d_7901_4a14_a4af_46d2582ccc5c.slice/crio-faa9d185fdd080afa6ebb5daae76e0ee73cdbc0f600922f6d4e75e94f18b3a87 WatchSource:0}: Error finding container faa9d185fdd080afa6ebb5daae76e0ee73cdbc0f600922f6d4e75e94f18b3a87: Status 404 returned error can't find the container with id faa9d185fdd080afa6ebb5daae76e0ee73cdbc0f600922f6d4e75e94f18b3a87 Nov 03 22:06:42.905718 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.905699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:06:42.912719 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.912696 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:06:42.914690 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:42.914667 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fb35fa_b562_4179_b2c5_9d8e3976bf2d.slice/crio-0bb5b0538a641eef684f65d2de9ed7ce71d94c8a15e307da2da9eba1c9139ab5 WatchSource:0}: Error finding container 0bb5b0538a641eef684f65d2de9ed7ce71d94c8a15e307da2da9eba1c9139ab5: Status 404 returned error can't find the container with id 0bb5b0538a641eef684f65d2de9ed7ce71d94c8a15e307da2da9eba1c9139ab5 Nov 03 22:06:42.916770 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:42.916753 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:06:42.920573 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:42.920552 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355e7f6f_eb2e_47a9_b4f3_0f7cfd5cb970.slice/crio-6b84b3417900acb306503f87c22b359da1caaab609ba89f8472ad54484c11278 WatchSource:0}: Error finding container 6b84b3417900acb306503f87c22b359da1caaab609ba89f8472ad54484c11278: Status 404 returned error can't find the container with id 6b84b3417900acb306503f87c22b359da1caaab609ba89f8472ad54484c11278 Nov 03 22:06:42.925725 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:06:42.925707 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c86199_dd69_41a7_84c6_fd3725cfa9f1.slice/crio-b55817bda055bd396d0500ade254739abe9045140f0c9e239d6b02286d133066 WatchSource:0}: Error finding container b55817bda055bd396d0500ade254739abe9045140f0c9e239d6b02286d133066: Status 404 returned error can't find the container with id b55817bda055bd396d0500ade254739abe9045140f0c9e239d6b02286d133066 Nov 03 22:06:43.276249 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.276180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:43.276249 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.276234 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcfj\" (UniqueName: \"kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj\") pod \"network-check-target-bljnw\" (UID: \"6b06086c-7464-4be0-a0c6-7568cb1dcd30\") " pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:43.276437 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:43.276332 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:43.276437 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.276348 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:43.276437 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:43.276358 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:43.276437 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:43.276375 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:43.276437 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:43.276384 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xwcfj for pod openshift-network-diagnostics/network-check-target-bljnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:43.276437 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:43.276387 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs podName:77d5bb8c-19f0-4d74-b146-ae9d69aebcdf nodeName:}" failed. No retries permitted until 2026-11-03 22:06:44.276373065 +0000 UTC m=+4.174997983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs") pod "network-metrics-daemon-4m4qq" (UID: "77d5bb8c-19f0-4d74-b146-ae9d69aebcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:43.276437 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:43.276437 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj podName:6b06086c-7464-4be0-a0c6-7568cb1dcd30 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:44.276425803 +0000 UTC m=+4.175050718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwcfj" (UniqueName: "kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj") pod "network-check-target-bljnw" (UID: "6b06086c-7464-4be0-a0c6-7568cb1dcd30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:43.276437 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:43.276439 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:43.276705 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:43.276474 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret podName:a0475429-6780-4dd9-8185-199a83ce41f5 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:44.276463343 +0000 UTC m=+4.175088256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret") pod "global-pull-secret-syncer-7zmnf" (UID: "a0475429-6780-4dd9-8185-199a83ce41f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:43.591442 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.591364 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6q9pz" event={"ID":"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d","Type":"ContainerStarted","Data":"2603ad2f8efbb8703e80500f47b3dc486eb54bff35395fcdfa0c9883009e8286"} Nov 03 22:06:43.592893 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.592868 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r2k9j" event={"ID":"6b091fb4-183e-41e9-a408-84fd1ca8f184","Type":"ContainerStarted","Data":"94005349a5bfe1a9a5137505223eefe7d3ea3d7ccec33d0cb5c8dddab9eaf14e"} Nov 03 22:06:43.594176 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.594130 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" event={"ID":"7f8a5faf-6b8f-4340-8b4c-945f8d67e008","Type":"ContainerStarted","Data":"ed72365a778a55a8b992ac6c62788f88a0cc69ca43e0bdcc7e3e3bc37095e5dd"} Nov 03 22:06:43.595405 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.595383 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k98wl" event={"ID":"76b201e3-7c0b-4810-9baa-f6e054cf0826","Type":"ContainerStarted","Data":"c6094815b5ac2e90de4c5282bb5ccd036fb0758d0919f6ee44149a20a5f7bd31"} Nov 03 22:06:43.596947 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.596909 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" event={"ID":"153d28e6-5d08-4064-9c61-2dd05100b2d0","Type":"ContainerStarted","Data":"550f19f0fa67d96d43c1ab43cd2db377b82f2878eda402cf7a5d968cc7fcf04b"} Nov 03 22:06:43.598150 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.598123 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" event={"ID":"93c86199-dd69-41a7-84c6-fd3725cfa9f1","Type":"ContainerStarted","Data":"b55817bda055bd396d0500ade254739abe9045140f0c9e239d6b02286d133066"} Nov 03 22:06:43.599765 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.599722 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8hqj" event={"ID":"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970","Type":"ContainerStarted","Data":"6b84b3417900acb306503f87c22b359da1caaab609ba89f8472ad54484c11278"} Nov 03 22:06:43.601111 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.601090 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lr6gj" event={"ID":"22fb35fa-b562-4179-b2c5-9d8e3976bf2d","Type":"ContainerStarted","Data":"0bb5b0538a641eef684f65d2de9ed7ce71d94c8a15e307da2da9eba1c9139ab5"} Nov 03 22:06:43.602298 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.602280 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b94tl" event={"ID":"cc56646d-7901-4a14-a4af-46d2582ccc5c","Type":"ContainerStarted","Data":"faa9d185fdd080afa6ebb5daae76e0ee73cdbc0f600922f6d4e75e94f18b3a87"} Nov 03 22:06:43.657337 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.657278 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:01:42 +0000 UTC" deadline="2028-08-10 20:38:05.281975388 +0000 UTC" Nov 03 22:06:43.657337 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:43.657329 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15502h31m21.624649276s" Nov 03 22:06:44.284407 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:44.284369 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcfj\" (UniqueName: \"kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj\") pod \"network-check-target-bljnw\" (UID: \"6b06086c-7464-4be0-a0c6-7568cb1dcd30\") " pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:44.284595 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:44.284435 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:44.284595 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:44.284466 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:44.284722 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:44.284607 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:44.284722 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:44.284685 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs podName:77d5bb8c-19f0-4d74-b146-ae9d69aebcdf nodeName:}" failed. No retries permitted until 2026-11-03 22:06:46.284666423 +0000 UTC m=+6.183291350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs") pod "network-metrics-daemon-4m4qq" (UID: "77d5bb8c-19f0-4d74-b146-ae9d69aebcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:44.284841 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:44.284779 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:44.284841 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:44.284794 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:44.284841 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:44.284808 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xwcfj for pod openshift-network-diagnostics/network-check-target-bljnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:44.284971 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:44.284842 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj podName:6b06086c-7464-4be0-a0c6-7568cb1dcd30 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:46.284831548 +0000 UTC m=+6.183456476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwcfj" (UniqueName: "kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj") pod "network-check-target-bljnw" (UID: "6b06086c-7464-4be0-a0c6-7568cb1dcd30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:44.284971 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:44.284900 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:44.284971 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:44.284934 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret podName:a0475429-6780-4dd9-8185-199a83ce41f5 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:46.284923622 +0000 UTC m=+6.183548540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret") pod "global-pull-secret-syncer-7zmnf" (UID: "a0475429-6780-4dd9-8185-199a83ce41f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:44.586254 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:44.585381 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:44.586254 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:44.585557 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:06:44.586254 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:44.585916 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:44.586254 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:44.585997 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:06:44.586254 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:44.586064 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:44.586254 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:44.586155 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:06:44.658499 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:44.658459 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:01:42 +0000 UTC" deadline="2028-07-27 02:59:07.797278142 +0000 UTC" Nov 03 22:06:44.658499 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:44.658497 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15148h52m23.138786117s" Nov 03 22:06:46.303691 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:46.303358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcfj\" (UniqueName: \"kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj\") pod \"network-check-target-bljnw\" (UID: \"6b06086c-7464-4be0-a0c6-7568cb1dcd30\") " pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:46.303691 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:46.303418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:46.303691 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:46.303451 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:46.303691 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:46.303517 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:46.303691 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:46.303535 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:46.303691 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:46.303549 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xwcfj for pod openshift-network-diagnostics/network-check-target-bljnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:46.303691 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:46.303616 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:46.303691 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:46.303652 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:46.303691 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:46.303621 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj podName:6b06086c-7464-4be0-a0c6-7568cb1dcd30 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:50.303601666 +0000 UTC m=+10.202226594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwcfj" (UniqueName: "kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj") pod "network-check-target-bljnw" (UID: "6b06086c-7464-4be0-a0c6-7568cb1dcd30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:46.304721 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:46.303726 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs podName:77d5bb8c-19f0-4d74-b146-ae9d69aebcdf nodeName:}" failed. No retries permitted until 2026-11-03 22:06:50.303707597 +0000 UTC m=+10.202332520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs") pod "network-metrics-daemon-4m4qq" (UID: "77d5bb8c-19f0-4d74-b146-ae9d69aebcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:46.304721 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:46.303742 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret podName:a0475429-6780-4dd9-8185-199a83ce41f5 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:50.303732735 +0000 UTC m=+10.202357653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret") pod "global-pull-secret-syncer-7zmnf" (UID: "a0475429-6780-4dd9-8185-199a83ce41f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:46.584015 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:46.583794 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:46.584160 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:46.583811 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:46.584160 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:46.584127 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:06:46.584160 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:46.583849 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:46.584313 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:46.584166 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:06:46.584313 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:46.584244 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:06:46.637911 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:46.637878 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" event={"ID":"815f1999fc2bcba4fa27ffca21498e4f","Type":"ContainerStarted","Data":"a370575ad259627e29d20536603aab751df0953decd8750649b88ffe21bafcb2"} Nov 03 22:06:46.656256 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:46.656223 2581 generic.go:356] "Generic (PLEG): container finished" podID="00032b9e4863070a848e5aad12046474" containerID="5ab761798d231fb90e25afcb65f7f11d7028007cd372627687c7b82ba6149211" exitCode=0 Nov 03 22:06:46.656376 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:46.656277 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" event={"ID":"00032b9e4863070a848e5aad12046474","Type":"ContainerDied","Data":"5ab761798d231fb90e25afcb65f7f11d7028007cd372627687c7b82ba6149211"} Nov 03 22:06:46.862883 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:46.862811 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" podStartSLOduration=4.862791276 podStartE2EDuration="4.862791276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:46.751008319 +0000 UTC m=+6.649633256" watchObservedRunningTime="2026-11-03 22:06:46.862791276 +0000 UTC m=+6.761416213" Nov 03 22:06:47.667330 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:47.666259 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" event={"ID":"00032b9e4863070a848e5aad12046474","Type":"ContainerStarted","Data":"b3a3296e32c3c46ad8a3b663d41c4c4fdbb6f1402cfff07ae1153d21edce4c16"} Nov 03 22:06:47.670814 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:47.670062 2581 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="ee1d7ca2921e7ed19a7da63561ff9584710ce6a08cc42b590ace5b87d69cea9a" exitCode=0 Nov 03 22:06:47.670814 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:47.670162 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8hqj" event={"ID":"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970","Type":"ContainerDied","Data":"ee1d7ca2921e7ed19a7da63561ff9584710ce6a08cc42b590ace5b87d69cea9a"} Nov 03 22:06:47.678459 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:47.678381 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b94tl" event={"ID":"cc56646d-7901-4a14-a4af-46d2582ccc5c","Type":"ContainerStarted","Data":"17d4feab201b31331edbfc9c1a19e77c920b3fcf603d9e096fbb1505514d4b1b"} Nov 03 22:06:47.897693 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:47.897581 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" podStartSLOduration=6.897560118 podStartE2EDuration="6.897560118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:47.785619971 +0000 UTC m=+7.684244907" watchObservedRunningTime="2026-11-03 22:06:47.897560118 +0000 UTC m=+7.796185054" Nov 03 22:06:48.021012 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:48.020949 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b94tl" podStartSLOduration=2.459728979 podStartE2EDuration="7.020931196s" totalImagesPullingTime="4.561202217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:47.898802248 +0000 UTC m=+7.797427177" watchObservedRunningTime="2026-11-03 22:06:48.020931196 +0000 UTC m=+7.919556133" Nov 03 22:06:48.584790 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:48.584756 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:48.584951 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:48.584890 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:06:48.584951 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:48.584931 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:48.585072 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:48.585011 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:06:48.585072 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:48.585053 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:48.585184 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:48.585114 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:06:49.684767 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:49.684730 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k98wl" event={"ID":"76b201e3-7c0b-4810-9baa-f6e054cf0826","Type":"ContainerStarted","Data":"fbe68a72a8e80e1b2455db78711ace864b8b0a631b94596fcae011b56230e188"} Nov 03 22:06:49.687205 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:49.687173 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" event={"ID":"93c86199-dd69-41a7-84c6-fd3725cfa9f1","Type":"ContainerStarted","Data":"d7e168e5b29cb84c8e3081a3e1caaf43b6e1aff00a156c5a4a46face74bd25c2"} Nov 03 22:06:49.722903 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:49.722272 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:06:49.723074 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:49.722943 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:06:49.916291 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:49.916180 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k98wl" podStartSLOduration=3.0332156 podStartE2EDuration="8.916159881s" totalImagesPullingTime="5.882944281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:49.803350179 +0000 UTC m=+9.701975116" watchObservedRunningTime="2026-11-03 22:06:49.916159881 +0000 UTC m=+9.814784818" Nov 03 22:06:50.337900 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:50.337809 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcfj\" (UniqueName: \"kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj\") pod \"network-check-target-bljnw\" (UID: \"6b06086c-7464-4be0-a0c6-7568cb1dcd30\") " pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:50.337900 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:50.337880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:50.338116 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:50.337924 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:50.338116 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:50.338097 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:50.338215 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:50.338159 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs podName:77d5bb8c-19f0-4d74-b146-ae9d69aebcdf nodeName:}" failed. No retries permitted until 2026-11-03 22:06:58.338141062 +0000 UTC m=+18.236765990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs") pod "network-metrics-daemon-4m4qq" (UID: "77d5bb8c-19f0-4d74-b146-ae9d69aebcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:50.338279 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:50.338241 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:50.338279 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:50.338256 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:50.338279 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:50.338266 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xwcfj for pod openshift-network-diagnostics/network-check-target-bljnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:50.338428 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:50.338290 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj podName:6b06086c-7464-4be0-a0c6-7568cb1dcd30 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:58.338282876 +0000 UTC m=+18.236907803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwcfj" (UniqueName: "kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj") pod "network-check-target-bljnw" (UID: "6b06086c-7464-4be0-a0c6-7568cb1dcd30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:50.338428 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:50.338327 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:50.338428 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:50.338363 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret podName:a0475429-6780-4dd9-8185-199a83ce41f5 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:58.338343832 +0000 UTC m=+18.236968760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret") pod "global-pull-secret-syncer-7zmnf" (UID: "a0475429-6780-4dd9-8185-199a83ce41f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:50.585761 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:50.585727 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:50.585928 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:50.585855 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:06:50.586260 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:50.586229 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:50.586371 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:50.586341 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:06:50.586408 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:50.586389 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:50.586492 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:50.586471 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:06:51.697443 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:51.696691 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r2k9j" event={"ID":"6b091fb4-183e-41e9-a408-84fd1ca8f184","Type":"ContainerStarted","Data":"4d57a1f761995274bdd51b67af3695a7766b825871258f4fa2f05167bfc00dd7"} Nov 03 22:06:51.791772 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:51.791736 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:06:52.518980 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.518811 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:06:51.791763773Z","UUID":"200cc438-ddac-41f4-ab91-475fda1595c6","Handler":null,"Name":"","Endpoint":""} Nov 03 22:06:52.520704 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.520681 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:06:52.520824 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.520712 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:06:52.584728 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.584698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:52.584869 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.584815 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:52.584869 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.584831 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:52.584989 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:52.584964 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:06:52.585075 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:52.585054 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:06:52.585500 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:52.585475 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:06:52.700171 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.700134 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6q9pz" event={"ID":"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d","Type":"ContainerStarted","Data":"710b362cdaad31fc3296c9cd6a7d79af94d7aaf86a326733158795258ebf1ca0"} Nov 03 22:06:52.706322 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.706288 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" event={"ID":"93c86199-dd69-41a7-84c6-fd3725cfa9f1","Type":"ContainerStarted","Data":"6d6629e730098cdfb947e37fe3b3adaa99acad0a476b045060ab9fa8ec76a5f4"} Nov 03 22:06:52.708514 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.708487 2581 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="f0d9aa308458cb37754acaa469bd713b19aad89298e7cbeb2f03161ccfabc0e3" exitCode=0 Nov 03 22:06:52.708684 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.708570 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8hqj" event={"ID":"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970","Type":"ContainerDied","Data":"f0d9aa308458cb37754acaa469bd713b19aad89298e7cbeb2f03161ccfabc0e3"} Nov 03 22:06:52.816608 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.816505 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-r2k9j" podStartSLOduration=3.399385285 podStartE2EDuration="11.816488042s" totalImagesPullingTime="8.417102757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:51.813172748 +0000 UTC m=+11.711797685" watchObservedRunningTime="2026-11-03 22:06:52.816488042 +0000 UTC m=+12.715112981" Nov 03 22:06:52.816787 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:52.816628 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6q9pz" podStartSLOduration=2.819223368 podStartE2EDuration="11.816623818s" totalImagesPullingTime="8.99740045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:52.815800596 +0000 UTC m=+12.714425533" watchObservedRunningTime="2026-11-03 22:06:52.816623818 +0000 UTC m=+12.715248753" Nov 03 22:06:53.713904 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:53.713567 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" event={"ID":"93c86199-dd69-41a7-84c6-fd3725cfa9f1","Type":"ContainerStarted","Data":"5dc3b295e3e78ab67da416b15d8803e49f9b75efbddb071aad14bb5af4969bc9"} Nov 03 22:06:53.715674 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:53.715625 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lr6gj" event={"ID":"22fb35fa-b562-4179-b2c5-9d8e3976bf2d","Type":"ContainerStarted","Data":"54110975afb04d6f31df73750af33dfe7d0ee8410857e7c517f9fc041122e62e"} Nov 03 22:06:53.717031 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:53.717006 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" event={"ID":"7f8a5faf-6b8f-4340-8b4c-945f8d67e008","Type":"ContainerStarted","Data":"efadcb57d6f643914b40cd1d596934476866463bb36920d9d4ae503a341d5c9b"} Nov 03 22:06:53.827812 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:53.827761 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" podStartSLOduration=2.704720136 podStartE2EDuration="12.82774393s" totalImagesPullingTime="10.123023794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:53.827557571 +0000 UTC m=+13.726182508" watchObservedRunningTime="2026-11-03 22:06:53.82774393 +0000 UTC m=+13.726368871" Nov 03 22:06:53.941930 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:53.941878 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" podStartSLOduration=2.328536874 podStartE2EDuration="12.941862867s" totalImagesPullingTime="10.613325993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:53.941597725 +0000 UTC m=+13.840222655" watchObservedRunningTime="2026-11-03 22:06:53.941862867 +0000 UTC m=+13.840487803" Nov 03 22:06:54.055677 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:54.055570 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lr6gj" podStartSLOduration=4.326258675 podStartE2EDuration="13.055553662s" totalImagesPullingTime="8.729294987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:54.054909305 +0000 UTC m=+13.953534243" watchObservedRunningTime="2026-11-03 22:06:54.055553662 +0000 UTC m=+13.954178598" Nov 03 22:06:54.583473 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:54.583400 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:54.583624 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:54.583414 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:54.583624 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:54.583518 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:54.583624 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:54.583512 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:06:54.583624 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:54.583609 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:06:54.583819 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:54.583731 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:06:54.722232 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:54.722199 2581 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="88414a89a81722716e13394ddc72bab71f1726b8135666b41833b7e82eec1417" exitCode=0 Nov 03 22:06:54.722812 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:54.722303 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8hqj" event={"ID":"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970","Type":"ContainerDied","Data":"88414a89a81722716e13394ddc72bab71f1726b8135666b41833b7e82eec1417"} Nov 03 22:06:56.582838 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:56.582802 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:56.582838 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:56.582803 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:56.583639 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:56.582821 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:56.583639 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:56.582943 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:06:56.583639 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:56.583039 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:06:56.583639 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:56.583084 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:06:56.729327 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:56.729247 2581 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="99dbb9b42ce4464fdf078eb2116b066256634d235b859f8892906ee2733410f0" exitCode=0 Nov 03 22:06:56.729327 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:56.729310 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8hqj" event={"ID":"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970","Type":"ContainerDied","Data":"99dbb9b42ce4464fdf078eb2116b066256634d235b859f8892906ee2733410f0"} Nov 03 22:06:58.392915 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:58.392631 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:58.393335 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:58.392769 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:58.393335 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:58.392953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:58.393335 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:58.393085 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret podName:a0475429-6780-4dd9-8185-199a83ce41f5 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:14.393064294 +0000 UTC m=+34.291689221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret") pod "global-pull-secret-syncer-7zmnf" (UID: "a0475429-6780-4dd9-8185-199a83ce41f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:58.393335 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:58.393166 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:58.393335 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:58.393219 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs podName:77d5bb8c-19f0-4d74-b146-ae9d69aebcdf nodeName:}" failed. No retries permitted until 2026-11-03 22:07:14.393203699 +0000 UTC m=+34.291828631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs") pod "network-metrics-daemon-4m4qq" (UID: "77d5bb8c-19f0-4d74-b146-ae9d69aebcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:58.393335 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:58.393163 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcfj\" (UniqueName: \"kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj\") pod \"network-check-target-bljnw\" (UID: \"6b06086c-7464-4be0-a0c6-7568cb1dcd30\") " pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:58.393335 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:58.393259 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:58.393335 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:58.393278 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:58.393335 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:58.393290 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xwcfj for pod openshift-network-diagnostics/network-check-target-bljnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:58.393335 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:58.393333 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj podName:6b06086c-7464-4be0-a0c6-7568cb1dcd30 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:14.393321224 +0000 UTC m=+34.291946143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwcfj" (UniqueName: "kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj") pod "network-check-target-bljnw" (UID: "6b06086c-7464-4be0-a0c6-7568cb1dcd30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:58.583556 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:58.583518 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:06:58.583714 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:58.583525 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:06:58.583714 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:58.583684 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:06:58.583714 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:58.583676 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:06:58.583890 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:58.583749 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:06:58.583890 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:06:58.583817 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:06:58.736863 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:58.736820 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" event={"ID":"153d28e6-5d08-4064-9c61-2dd05100b2d0","Type":"ContainerStarted","Data":"0e71a2f73477b6edd8cada5163f94262460f29d7ccbda32779324c4c24757b5e"} Nov 03 22:06:58.737015 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:58.736912 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" event={"ID":"153d28e6-5d08-4064-9c61-2dd05100b2d0","Type":"ContainerStarted","Data":"3c989f8ff8bf2fc9141c516685bbcba4827ce8ecce83efa71142062511dcad45"} Nov 03 22:06:58.737015 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:58.736966 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" event={"ID":"153d28e6-5d08-4064-9c61-2dd05100b2d0","Type":"ContainerStarted","Data":"2b74476faf75e6f32d8af06c8dbf1af1e0fbbd902ffe10c5a1ab07e944bc31a9"} Nov 03 22:06:58.737015 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:58.737008 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" event={"ID":"153d28e6-5d08-4064-9c61-2dd05100b2d0","Type":"ContainerStarted","Data":"9334a65f5990871b299a5c7c78144d08d437cfa73dd13241d6e95eeaa57fb10f"} Nov 03 22:06:58.737135 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:58.737051 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" event={"ID":"153d28e6-5d08-4064-9c61-2dd05100b2d0","Type":"ContainerStarted","Data":"902f4c469b2228637e6f65dd10429bfcbfa160d121badc1bf7586f09bffd7ff0"} Nov 03 22:06:59.741460 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:59.741276 2581 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="881c2e2243b5d7e0a262ad4bedca718cdb9430913e1783d029536bb403d9ebfa" exitCode=0 Nov 03 22:06:59.741951 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:59.741360 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8hqj" event={"ID":"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970","Type":"ContainerDied","Data":"881c2e2243b5d7e0a262ad4bedca718cdb9430913e1783d029536bb403d9ebfa"} Nov 03 22:06:59.746136 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:06:59.746111 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" event={"ID":"153d28e6-5d08-4064-9c61-2dd05100b2d0","Type":"ContainerStarted","Data":"16d406180d3758fa075ac32e8240934e1f58e03b99bd90dfaae4abae01abcf32"} Nov 03 22:07:00.583819 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:00.583778 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:00.583991 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:00.583831 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:00.583991 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:00.583912 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:07:00.583991 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:00.583955 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:07:00.584099 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:00.583998 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:00.584099 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:00.584084 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:07:00.751428 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:00.751392 2581 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="3be48978eb05ab493fc25644ed368bb31f7d896a6731e6993edd85dab8e72f93" exitCode=0 Nov 03 22:07:00.751428 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:00.751431 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8hqj" event={"ID":"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970","Type":"ContainerDied","Data":"3be48978eb05ab493fc25644ed368bb31f7d896a6731e6993edd85dab8e72f93"} Nov 03 22:07:01.757173 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:01.757139 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8hqj" event={"ID":"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970","Type":"ContainerStarted","Data":"0ce45d42197081bc080045fa5085711b45b810875c66b16d0cc9be3075a6554d"} Nov 03 22:07:01.760223 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:01.760198 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" event={"ID":"153d28e6-5d08-4064-9c61-2dd05100b2d0","Type":"ContainerStarted","Data":"5737add959acd02e2d4408fbab79a59190afa41928698c74a6fb8bfb33655cea"} Nov 03 22:07:01.875598 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:01.875538 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f8hqj" podStartSLOduration=7.417816092 podStartE2EDuration="20.87552569s" totalImagesPullingTime="13.457709598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:01.874981132 +0000 UTC m=+21.773606067" watchObservedRunningTime="2026-11-03 22:07:01.87552569 +0000 UTC m=+21.774150635" Nov 03 22:07:02.132225 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:02.132148 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:02.358609 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:02.358577 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:02.582765 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:02.582681 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:02.582765 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:02.582698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:02.582970 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:02.582788 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:02.582970 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:02.582792 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:07:02.582970 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:02.582866 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:07:02.582970 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:02.582940 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:07:03.768277 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:03.768242 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" event={"ID":"153d28e6-5d08-4064-9c61-2dd05100b2d0","Type":"ContainerStarted","Data":"c16b0d6c45855091066a9fc252cd0411afd6d9133e215759926be7a62a4b75bd"} Nov 03 22:07:03.768993 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:03.768544 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:03.768993 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:03.768604 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:03.783631 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:03.783605 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:03.881275 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:03.881245 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:07:03.881426 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:03.881361 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:07:03.881910 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:03.881889 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:07:04.025708 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:04.025597 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" podStartSLOduration=7.564128724 podStartE2EDuration="23.025585513s" totalImagesPullingTime="15.461456789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:04.025326212 +0000 UTC m=+23.923951159" watchObservedRunningTime="2026-11-03 22:07:04.025585513 +0000 UTC m=+23.924210450" Nov 03 22:07:04.247534 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:04.247505 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:04.586295 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:04.586261 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:04.586487 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:04.586361 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:04.586487 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:04.586379 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:07:04.586487 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:04.586450 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:07:04.586487 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:04.586483 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:04.586690 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:04.586529 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:07:04.774615 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:04.771954 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:04.786936 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:04.786910 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:05.582466 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:05.582346 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7zmnf"] Nov 03 22:07:05.582668 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:05.582480 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:05.582668 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:05.582579 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:07:05.585809 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:05.585781 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bljnw"] Nov 03 22:07:05.585914 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:05.585873 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:05.585970 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:05.585949 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:07:05.586681 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:05.586659 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4m4qq"] Nov 03 22:07:05.586793 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:05.586778 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:05.586911 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:05.586890 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:07:05.791986 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:05.791956 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:05.960968 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:05.960894 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:07.583112 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:07.583079 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:07.583462 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:07.583119 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:07.583462 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:07.583118 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:07.583462 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:07.583251 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:07:07.583462 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:07.583230 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:07:07.583462 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:07.583333 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:07:08.737006 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:08.736963 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:08.908924 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:08.908860 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:09.082843 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:09.082765 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:09.260791 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:09.260760 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:09.582763 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:09.582692 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:09.582763 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:09.582720 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:09.582926 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:09.582804 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m4qq" podUID="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" Nov 03 22:07:09.582926 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:09.582857 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:09.582993 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:09.582931 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bljnw" podUID="6b06086c-7464-4be0-a0c6-7568cb1dcd30" Nov 03 22:07:09.583042 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:09.583016 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zmnf" podUID="a0475429-6780-4dd9-8185-199a83ce41f5" Nov 03 22:07:09.707189 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:09.707158 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:09.901122 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:09.901038 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:10.405296 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:10.405268 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:10.592952 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:10.592920 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:11.017302 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.017273 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:11.198570 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.198486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:11.461697 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.461618 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeReady" Nov 03 22:07:11.461834 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.461767 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 22:07:11.583134 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.583098 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:11.583306 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.583145 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:11.583306 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.583198 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:11.632729 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.632694 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:07:11.632887 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.632764 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:07:11.632887 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.632775 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ng2g2\"" Nov 03 22:07:11.632887 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.632817 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:07:11.632887 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.632872 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:07:11.633020 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.632939 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g6ff6\"" Nov 03 22:07:11.633020 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.632988 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kxpq\"" Nov 03 22:07:11.879906 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:11.879822 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:12.053844 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:12.053808 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:12.444329 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:12.444298 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6q9pz_d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d/dns-node-resolver/0.log" Nov 03 22:07:12.448938 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:12.448910 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:12.875132 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:12.875095 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:13.049279 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:13.049250 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:13.368173 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:13.368145 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:13.445481 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:13.445451 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k98wl_76b201e3-7c0b-4810-9baa-f6e054cf0826/node-ca/0.log" Nov 03 22:07:13.611302 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:13.611271 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:13.988238 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:13.988210 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:14.161262 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:14.161229 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:14.410683 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:14.410570 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcfj\" (UniqueName: \"kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj\") pod \"network-check-target-bljnw\" (UID: \"6b06086c-7464-4be0-a0c6-7568cb1dcd30\") " pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:14.410683 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:14.410627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:14.410683 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:14.410675 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:14.411217 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:14.410801 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:07:14.411217 ip-10-0-139-254 kubenswrapper[2581]: E1103 22:07:14.410876 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs podName:77d5bb8c-19f0-4d74-b146-ae9d69aebcdf nodeName:}" failed. No retries permitted until 2026-11-03 22:07:46.410856801 +0000 UTC m=+66.309481732 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs") pod "network-metrics-daemon-4m4qq" (UID: "77d5bb8c-19f0-4d74-b146-ae9d69aebcdf") : secret "metrics-daemon-secret" not found Nov 03 22:07:14.413955 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:14.413930 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:14.414192 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:14.414173 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwcfj\" (UniqueName: \"kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj\") pod \"network-check-target-bljnw\" (UID: \"6b06086c-7464-4be0-a0c6-7568cb1dcd30\") " pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:14.550014 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:14.549982 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:14.645140 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:14.645105 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:14.657444 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:14.657413 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:14.730225 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:14.730198 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:14.902973 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:14.902939 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:15.071149 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:15.071114 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:15.098700 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:15.098667 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7zmnf"] Nov 03 22:07:15.102128 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:07:15.102102 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0475429_6780_4dd9_8185_199a83ce41f5.slice/crio-188d93f62ce34515ada60b28aeec83adf435822944c8f12b7bcfedf56a1366ec WatchSource:0}: Error finding container 188d93f62ce34515ada60b28aeec83adf435822944c8f12b7bcfedf56a1366ec: Status 404 returned error can't find the container with id 188d93f62ce34515ada60b28aeec83adf435822944c8f12b7bcfedf56a1366ec Nov 03 22:07:15.146151 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:15.146123 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bljnw"] Nov 03 22:07:15.148976 ip-10-0-139-254 kubenswrapper[2581]: W1103 22:07:15.148950 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b06086c_7464_4be0_a0c6_7568cb1dcd30.slice/crio-10d481d4cd6bf34ce4eb7f990c6072a5493321b75cfc6bf48672cc526a9258fc WatchSource:0}: Error finding container 10d481d4cd6bf34ce4eb7f990c6072a5493321b75cfc6bf48672cc526a9258fc: Status 404 returned error can't find the container with id 10d481d4cd6bf34ce4eb7f990c6072a5493321b75cfc6bf48672cc526a9258fc Nov 03 22:07:15.353624 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:15.353545 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:15.538046 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:15.538016 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:15.717862 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:15.717828 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:15.796906 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:15.796870 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bljnw" event={"ID":"6b06086c-7464-4be0-a0c6-7568cb1dcd30","Type":"ContainerStarted","Data":"10d481d4cd6bf34ce4eb7f990c6072a5493321b75cfc6bf48672cc526a9258fc"} Nov 03 22:07:15.797979 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:15.797947 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7zmnf" event={"ID":"a0475429-6780-4dd9-8185-199a83ce41f5","Type":"ContainerStarted","Data":"188d93f62ce34515ada60b28aeec83adf435822944c8f12b7bcfedf56a1366ec"} Nov 03 22:07:15.893742 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:15.893707 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:16.057518 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:16.057440 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:16.226753 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:16.226721 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:16.556270 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:16.556227 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:16.750410 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:16.750364 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:16.961507 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:16.961474 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:17.131718 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:17.131678 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:17.356294 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:17.356209 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:17.542614 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:17.542568 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:17.724519 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:17.724483 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:18.151092 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:18.151011 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:18.553790 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:18.553753 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:18.752679 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:18.752631 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:18.990327 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:18.990296 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:19.356850 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:19.356823 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:19.809841 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:19.809806 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bljnw" event={"ID":"6b06086c-7464-4be0-a0c6-7568cb1dcd30","Type":"ContainerStarted","Data":"2d65d055e590c876889433db664c06094603a8148ac1549983c28fb855ec9ecd"} Nov 03 22:07:19.810259 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:19.809905 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:19.923606 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:19.923545 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bljnw" podStartSLOduration=34.741294957 podStartE2EDuration="38.923527756s" totalImagesPullingTime="4.182232799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:19.922818149 +0000 UTC m=+39.821443086" watchObservedRunningTime="2026-11-03 22:07:19.923527756 +0000 UTC m=+39.822152693" Nov 03 22:07:20.164149 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:20.163996 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:20.560163 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:20.560130 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:20.759720 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:20.759679 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:20.959238 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:20.959203 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:21.360150 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:21.360070 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:22.155958 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:22.155919 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:22.548568 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:22.548524 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:22.753567 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:22.753542 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:22.821002 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:22.820914 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7zmnf" event={"ID":"a0475429-6780-4dd9-8185-199a83ce41f5","Type":"ContainerStarted","Data":"ffa172b6eda3eea7d159191620d7dcaafee687da4f84eed5792c85ec9277138f"} Nov 03 22:07:22.934360 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:22.934313 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7zmnf" podStartSLOduration=34.374838849 podStartE2EDuration="41.934298912s" totalImagesPullingTime="7.559460063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:22.933916422 +0000 UTC m=+42.832541361" watchObservedRunningTime="2026-11-03 22:07:22.934298912 +0000 UTC m=+42.832923847" Nov 03 22:07:22.956893 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:22.956866 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:22.958381 ip-10-0-139-254 kubenswrapper[2581]: I1103 22:07:22.958343 2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:07:22.958484 ip-10-0-139-254 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 22:07:22.977111 ip-10-0-139-254 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 22:07:22.977285 ip-10-0-139-254 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 22:07:22.977337 ip-10-0-139-254 systemd[1]: kubelet.service: Consumed 2.392s CPU time. Nov 03 22:07:22.989194 ip-10-0-139-254 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:07:23.048565 ip-10-0-139-254 kubenswrapper[4298]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:07:23.048565 ip-10-0-139-254 kubenswrapper[4298]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:07:23.048565 ip-10-0-139-254 kubenswrapper[4298]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:07:23.048565 ip-10-0-139-254 kubenswrapper[4298]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:07:23.050702 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050591 4298 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:23.050702 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050701 4298 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:23.050702 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050706 4298 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050709 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050712 4298 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050714 4298 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050719 4298 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050721 4298 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050724 4298 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050727 4298 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050730 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050732 4298 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050735 4298 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050737 4298 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050740 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050743 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050745 4298 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050748 4298 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050751 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050753 4298 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050756 4298 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050759 4298 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:23.050805 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050761 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050764 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050767 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050770 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050773 4298 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050775 4298 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050778 4298 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050780 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050783 4298 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050785 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050788 4298 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050791 4298 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050793 4298 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050796 4298 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050799 4298 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050802 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050805 4298 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050808 4298 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050811 4298 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:23.051383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050813 4298 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050816 4298 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050819 4298 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050821 4298 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050824 4298 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050827 4298 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050829 4298 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050832 4298 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050834 4298 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050837 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050839 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050843 4298 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050848 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050851 4298 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050855 4298 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050858 4298 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050860 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050863 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050866 4298 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:23.051871 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050868 4298 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050871 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050875 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050878 4298 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050881 4298 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050883 4298 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050886 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050888 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050892 4298 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050894 4298 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050897 4298 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050899 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050902 4298 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050905 4298 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050908 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050911 4298 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050913 4298 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050916 4298 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050919 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050921 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:23.052353 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050924 4298 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050926 4298 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050929 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050931 4298 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050934 4298 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050937 4298 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050939 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050942 4298 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050944 4298 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050947 4298 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050950 4298 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050952 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050955 4298 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050957 4298 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050960 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050963 4298 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050969 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050971 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050974 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:23.052859 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050976 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050979 4298 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050983 4298 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050985 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050988 4298 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050993 4298 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.050997 4298 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051776 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051783 4298 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051787 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051789 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051792 4298 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051795 4298 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051798 4298 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051801 4298 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051804 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051806 4298 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051809 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051812 4298 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051815 4298 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:23.053322 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051817 4298 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051820 4298 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051823 4298 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051825 4298 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051828 4298 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051830 4298 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051833 4298 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051836 4298 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051838 4298 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051842 4298 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051844 4298 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051847 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051850 4298 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051852 4298 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051856 4298 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051859 4298 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051861 4298 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051864 4298 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051866 4298 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:23.053842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051870 4298 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051873 4298 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051875 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051878 4298 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051881 4298 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051885 4298 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051890 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051893 4298 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051896 4298 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051899 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051901 4298 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051904 4298 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051906 4298 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051909 4298 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051912 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051914 4298 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051916 4298 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051919 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051922 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:23.054307 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051924 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051927 4298 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051929 4298 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051932 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051935 4298 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051937 4298 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051940 4298 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051943 4298 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051946 4298 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051949 4298 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051951 4298 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051954 4298 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051957 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051959 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051962 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051967 4298 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051971 4298 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051974 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051977 4298 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:23.054790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051979 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051982 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051985 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051987 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051990 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051992 4298 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051995 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.051997 4298 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052000 4298 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052002 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052005 4298 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052007 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052010 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052012 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052015 4298 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052018 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052021 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052023 4298 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052026 4298 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052029 4298 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:23.055256 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052031 4298 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052034 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052037 4298 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052039 4298 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052042 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052045 4298 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052047 4298 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052050 4298 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052053 4298 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052055 4298 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052059 4298 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052061 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052064 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052066 4298 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052069 4298 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052071 4298 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052576 4298 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052582 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052584 4298 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052588 4298 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:23.055791 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052590 4298 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052593 4298 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052597 4298 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052601 4298 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052605 4298 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052608 4298 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052611 4298 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052615 4298 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052618 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052620 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052624 4298 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052627 4298 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052630 4298 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052632 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052635 4298 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052657 4298 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052660 4298 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052663 4298 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052666 4298 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052668 4298 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:23.056275 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052672 4298 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052675 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052677 4298 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052680 4298 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052683 4298 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052685 4298 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052688 4298 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052690 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052693 4298 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052696 4298 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052698 4298 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052701 4298 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052703 4298 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052706 4298 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052708 4298 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052711 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052716 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052718 4298 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052721 4298 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052724 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:23.056820 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052727 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052729 4298 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052732 4298 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052735 4298 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052737 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052740 4298 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052743 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052745 4298 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052748 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052751 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052753 4298 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052756 4298 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052758 4298 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052761 4298 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052764 4298 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052766 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052769 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052772 4298 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052775 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:23.057317 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052778 4298 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052780 4298 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052783 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052785 4298 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052788 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052790 4298 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052793 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052795 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052798 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052802 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052805 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052808 4298 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052812 4298 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052816 4298 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052819 4298 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052822 4298 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052824 4298 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052827 4298 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052830 4298 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:23.057794 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052832 4298 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052835 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052837 4298 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052839 4298 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052842 4298 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052845 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052847 4298 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052850 4298 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052853 4298 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052856 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052858 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052861 4298 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052864 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052867 4298 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052870 4298 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052872 4298 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052875 4298 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052878 4298 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052881 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052884 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:23.058251 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052887 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052889 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052893 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.052895 4298 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053009 4298 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053021 4298 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053029 4298 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053034 4298 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053039 4298 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053042 4298 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053047 4298 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053051 4298 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053054 4298 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053059 4298 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053062 4298 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053065 4298 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053069 4298 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053072 4298 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053075 4298 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053078 4298 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053081 4298 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053084 4298 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053089 4298 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053092 4298 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:07:23.058735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053095 4298 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053098 4298 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053102 4298 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053106 4298 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053109 4298 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053113 4298 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053116 4298 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053119 4298 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053123 4298 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053126 4298 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053130 4298 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053134 4298 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053138 4298 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053141 4298 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053144 4298 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053147 4298 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053150 4298 flags.go:64] FLAG: --enable-server="true" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053153 4298 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053158 4298 flags.go:64] FLAG: --event-burst="100" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053161 4298 flags.go:64] FLAG: --event-qps="50" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053164 4298 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053167 4298 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053170 4298 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053174 4298 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:07:23.060393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053177 4298 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053180 4298 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053183 4298 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053188 4298 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053191 4298 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053194 4298 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053198 4298 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053201 4298 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053204 4298 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053207 4298 flags.go:64] FLAG: --feature-gates="" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053211 4298 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053214 4298 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053217 4298 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053220 4298 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053223 4298 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053227 4298 flags.go:64] FLAG: --help="false" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053230 4298 flags.go:64] FLAG: --hostname-override="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053233 4298 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053238 4298 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053241 4298 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053244 4298 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053248 4298 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053251 4298 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:07:23.061031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053253 4298 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053256 4298 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053259 4298 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053262 4298 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053265 4298 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053268 4298 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053271 4298 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053274 4298 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053277 4298 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053280 4298 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053283 4298 flags.go:64] FLAG: --lock-file="" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053286 4298 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053291 4298 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053294 4298 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053299 4298 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053303 4298 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053307 4298 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053310 4298 flags.go:64] FLAG: --logging-format="text" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053313 4298 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053316 4298 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053319 4298 flags.go:64] FLAG: --manifest-url="" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053322 4298 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053327 4298 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053330 4298 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053334 4298 flags.go:64] FLAG: --max-pods="110" Nov 03 22:07:23.061587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053338 4298 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053341 4298 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053345 4298 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053348 4298 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053352 4298 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053355 4298 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053358 4298 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053365 4298 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053368 4298 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053371 4298 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053374 4298 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053377 4298 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053380 4298 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053383 4298 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053386 4298 flags.go:64] FLAG: --port="10250" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053389 4298 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053392 4298 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-047375d5d7ca8f156" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053395 4298 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053399 4298 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053404 4298 flags.go:64] FLAG: --register-node="true" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053407 4298 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053411 4298 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053413 4298 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053416 4298 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053420 4298 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053424 4298 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:07:23.062301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053427 4298 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053430 4298 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053433 4298 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053436 4298 flags.go:64] FLAG: --runonce="false" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053439 4298 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053442 4298 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053445 4298 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053448 4298 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053453 4298 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053456 4298 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053459 4298 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053463 4298 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053465 4298 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053468 4298 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053471 4298 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053474 4298 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053478 4298 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053481 4298 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053483 4298 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053489 4298 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053492 4298 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053495 4298 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053499 4298 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053501 4298 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053504 4298 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:07:23.063013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053508 4298 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053511 4298 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053515 4298 flags.go:64] FLAG: --v="2" Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053519 4298 flags.go:64] FLAG: --version="false" Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053523 4298 flags.go:64] FLAG: --vmodule="" Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053527 4298 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.053531 4298 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053694 4298 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053700 4298 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053703 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053706 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053709 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053712 4298 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053716 4298 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053719 4298 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053723 4298 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053726 4298 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053728 4298 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053731 4298 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053734 4298 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053737 4298 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:23.063617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053739 4298 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053742 4298 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053745 4298 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053748 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053751 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053753 4298 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053756 4298 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053759 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053762 4298 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053764 4298 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053767 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053771 4298 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053774 4298 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053776 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053779 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053781 4298 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053784 4298 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053787 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053790 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:23.064148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053793 4298 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053796 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053798 4298 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053801 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053803 4298 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053806 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053809 4298 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053813 4298 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053816 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053818 4298 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053821 4298 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053823 4298 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053826 4298 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053829 4298 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053831 4298 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053834 4298 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053837 4298 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053840 4298 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053842 4298 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053845 4298 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:23.064727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053848 4298 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053851 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053853 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053856 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053860 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053863 4298 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053865 4298 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053868 4298 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053871 4298 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053874 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053876 4298 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053879 4298 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053882 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053885 4298 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053887 4298 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053890 4298 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053892 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053895 4298 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053898 4298 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053902 4298 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:23.065373 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053904 4298 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053907 4298 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053911 4298 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053914 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053917 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053920 4298 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053923 4298 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053925 4298 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053928 4298 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053931 4298 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053934 4298 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053936 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053939 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053942 4298 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053945 4298 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053947 4298 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053951 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053954 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053956 4298 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053959 4298 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:23.066138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053961 4298 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053965 4298 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053969 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053972 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053975 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053994 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.053997 4298 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054001 4298 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054004 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054006 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054009 4298 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054013 4298 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054015 4298 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.054044 4298 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054080 4298 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:23.067040 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054084 4298 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054087 4298 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054090 4298 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054094 4298 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054096 4298 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054099 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054102 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054104 4298 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054107 4298 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054110 4298 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054113 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054115 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054118 4298 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054122 4298 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054124 4298 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054129 4298 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054133 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054136 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054138 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054142 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:23.067617 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054145 4298 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054147 4298 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054150 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054153 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054156 4298 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054158 4298 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054161 4298 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054163 4298 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054166 4298 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054170 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054172 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054175 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054178 4298 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054180 4298 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054183 4298 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054185 4298 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054188 4298 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054190 4298 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054193 4298 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054196 4298 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:23.068193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054198 4298 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054201 4298 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054204 4298 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054206 4298 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054209 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054213 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054216 4298 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054218 4298 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054221 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054223 4298 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054226 4298 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054228 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054231 4298 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054234 4298 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054236 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054239 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054242 4298 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054245 4298 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054247 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:23.068708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054250 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054252 4298 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054256 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054259 4298 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054262 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054264 4298 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054267 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054270 4298 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054273 4298 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054276 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054278 4298 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054281 4298 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054283 4298 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054286 4298 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054288 4298 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054291 4298 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054294 4298 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054296 4298 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054300 4298 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054302 4298 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:23.069226 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054305 4298 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054307 4298 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054310 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054313 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054316 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054319 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054321 4298 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054324 4298 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054326 4298 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054329 4298 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054332 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054335 4298 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054338 4298 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054341 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054345 4298 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054347 4298 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054350 4298 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054353 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054355 4298 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054358 4298 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:23.069727 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054360 4298 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054363 4298 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054366 4298 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054370 4298 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054373 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.054376 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.058546 4298 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.058563 4298 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.058581 4298 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.058590 4298 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058676 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058683 4298 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058687 4298 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058699 4298 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058702 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058705 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058707 4298 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058710 4298 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:23.070213 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058713 4298 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058715 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058718 4298 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058721 4298 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058724 4298 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058727 4298 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058730 4298 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058733 4298 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058735 4298 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058738 4298 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058741 4298 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058744 4298 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058746 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058750 4298 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058753 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058756 4298 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058759 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058762 4298 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058764 4298 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058767 4298 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:23.070714 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058769 4298 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058778 4298 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058781 4298 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058784 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058786 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058789 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058791 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058794 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058796 4298 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058799 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058802 4298 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058804 4298 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058807 4298 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058810 4298 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058812 4298 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058815 4298 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058818 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058821 4298 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058823 4298 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058826 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:23.071232 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058828 4298 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058831 4298 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058834 4298 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058836 4298 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058839 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058842 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058844 4298 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058847 4298 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058850 4298 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058853 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058855 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058858 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058860 4298 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058863 4298 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058871 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058874 4298 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058877 4298 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058879 4298 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058882 4298 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058885 4298 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:23.071792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058887 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058890 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058893 4298 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058895 4298 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058898 4298 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058901 4298 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058903 4298 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058906 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058909 4298 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058911 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058914 4298 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058917 4298 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058921 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058924 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058926 4298 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058929 4298 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058932 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058935 4298 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:23.072271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058939 4298 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058943 4298 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058947 4298 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058951 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058954 4298 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058957 4298 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058960 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058963 4298 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058966 4298 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058974 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058977 4298 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058980 4298 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058982 4298 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058985 4298 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058988 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058990 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058993 4298 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058996 4298 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.058998 4298 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:23.072747 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059001 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.059028 4298 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059616 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059625 4298 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059629 4298 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059635 4298 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059637 4298 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059655 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059659 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059662 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059665 4298 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059668 4298 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059671 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059674 4298 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059677 4298 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059680 4298 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:23.073219 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059683 4298 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059685 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059691 4298 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059694 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059697 4298 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059699 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059702 4298 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059728 4298 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059731 4298 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059734 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059737 4298 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059740 4298 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059743 4298 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059745 4298 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059752 4298 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059754 4298 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059757 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059760 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059763 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:23.073605 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059766 4298 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059768 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059771 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059774 4298 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059776 4298 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059779 4298 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059782 4298 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059784 4298 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059790 4298 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059793 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059796 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059799 4298 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059801 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059804 4298 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059807 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059810 4298 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059812 4298 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059815 4298 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059818 4298 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059821 4298 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:23.074091 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059824 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059849 4298 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059852 4298 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059855 4298 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059858 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059861 4298 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059864 4298 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059866 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059869 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059871 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059875 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059877 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059880 4298 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059883 4298 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059888 4298 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059891 4298 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059894 4298 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059896 4298 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059899 4298 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:23.074587 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059902 4298 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059905 4298 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059908 4298 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059910 4298 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059915 4298 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059920 4298 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059923 4298 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059929 4298 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059932 4298 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059935 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059938 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059941 4298 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059944 4298 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059947 4298 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059950 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059973 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059977 4298 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059980 4298 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059982 4298 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:23.075074 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059988 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059991 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059994 4298 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059996 4298 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.059999 4298 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060002 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060004 4298 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060007 4298 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060011 4298 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060015 4298 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060018 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060021 4298 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060024 4298 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060030 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060032 4298 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.060067 4298 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:07:23.075526 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060164 4298 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060170 4298 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060173 4298 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060177 4298 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060180 4298 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060183 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060186 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060189 4298 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060192 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060195 4298 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060223 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060405 4298 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060419 4298 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060423 4298 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060427 4298 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060430 4298 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060433 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060436 4298 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060439 4298 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060441 4298 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:23.075919 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060445 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060447 4298 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060450 4298 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060453 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060456 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060458 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060463 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060466 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060469 4298 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060472 4298 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060475 4298 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060478 4298 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060482 4298 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060485 4298 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060488 4298 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060490 4298 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060493 4298 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060496 4298 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060499 4298 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:23.076449 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060505 4298 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060512 4298 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060515 4298 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060519 4298 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060521 4298 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060525 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060528 4298 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060531 4298 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060535 4298 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060537 4298 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060540 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060543 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060546 4298 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060548 4298 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060551 4298 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060554 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060556 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060559 4298 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060562 4298 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060564 4298 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:23.076976 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060567 4298 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060569 4298 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060572 4298 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060578 4298 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060582 4298 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060585 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060588 4298 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060591 4298 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060594 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060596 4298 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060599 4298 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060602 4298 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060605 4298 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060608 4298 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060611 4298 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060614 4298 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060617 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060620 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060623 4298 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:23.077751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060625 4298 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060628 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060631 4298 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060634 4298 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060637 4298 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060657 4298 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060662 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060664 4298 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060667 4298 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060670 4298 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060672 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060675 4298 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060678 4298 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060681 4298 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060683 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060686 4298 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060689 4298 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060691 4298 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060695 4298 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060697 4298 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:23.078448 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060700 4298 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060704 4298 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060707 4298 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060710 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060713 4298 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060716 4298 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060718 4298 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:23.060721 4298 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.060795 4298 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.061848 4298 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.061904 4298 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.062086 4298 server.go:1014] "Starting client certificate rotation" Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.062233 4298 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.062301 4298 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:01:40 +0000 UTC" deadline="2028-05-27 19:50:36.390362005 +0000 UTC" Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.062338 4298 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13701h43m13.328027359s" Nov 03 22:07:23.079271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.062550 4298 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:07:23.079820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.063703 4298 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:07:23.079820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.065255 4298 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:07:23.079820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.067047 4298 log.go:25] "Validated CRI v1 image API" Nov 03 22:07:23.079820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.067783 4298 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:07:23.079820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.069122 4298 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ca621f77-771f-4b8c-8a2b-690734981466:/dev/nvme0n1p4 f8a8aa0f-c912-477a-a7a2-18f39b7c48b1:/dev/nvme0n1p3] Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.069141 4298 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bb5b0538a641eef684f65d2de9ed7ce71d94c8a15e307da2da9eba1c9139ab5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bb5b0538a641eef684f65d2de9ed7ce71d94c8a15e307da2da9eba1c9139ab5/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10d481d4cd6bf34ce4eb7f990c6072a5493321b75cfc6bf48672cc526a9258fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10d481d4cd6bf34ce4eb7f990c6072a5493321b75cfc6bf48672cc526a9258fc/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/188d93f62ce34515ada60b28aeec83adf435822944c8f12b7bcfedf56a1366ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/188d93f62ce34515ada60b28aeec83adf435822944c8f12b7bcfedf56a1366ec/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2603ad2f8efbb8703e80500f47b3dc486eb54bff35395fcdfa0c9883009e8286/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2603ad2f8efbb8703e80500f47b3dc486eb54bff35395fcdfa0c9883009e8286/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/550f19f0fa67d96d43c1ab43cd2db377b82f2878eda402cf7a5d968cc7fcf04b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/550f19f0fa67d96d43c1ab43cd2db377b82f2878eda402cf7a5d968cc7fcf04b/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b84b3417900acb306503f87c22b359da1caaab609ba89f8472ad54484c11278/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b84b3417900acb306503f87c22b359da1caaab609ba89f8472ad54484c11278/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85f69c9fc91998bfb5d5cdf395dfedfd5d780b67180868f2909ad106762380be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85f69c9fc91998bfb5d5cdf395dfedfd5d780b67180868f2909ad106762380be/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ce713afc7eaf63a516c415eff23041ed2464eefebb24c6ca371432da39f7410/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ce713afc7eaf63a516c415eff23041ed2464eefebb24c6ca371432da39f7410/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94005349a5bfe1a9a5137505223eefe7d3ea3d7ccec33d0cb5c8dddab9eaf14e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94005349a5bfe1a9a5137505223eefe7d3ea3d7ccec33d0cb5c8dddab9eaf14e/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b55817bda055bd396d0500ade254739abe9045140f0c9e239d6b02286d133066/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b55817bda055bd396d0500ade254739abe9045140f0c9e239d6b02286d133066/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6094815b5ac2e90de4c5282bb5ccd036fb0758d0919f6ee44149a20a5f7bd31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6094815b5ac2e90de4c5282bb5ccd036fb0758d0919f6ee44149a20a5f7bd31/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/faa9d185fdd080afa6ebb5daae76e0ee73cdbc0f600922f6d4e75e94f18b3a87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/faa9d185fdd080afa6ebb5daae76e0ee73cdbc0f600922f6d4e75e94f18b3a87/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/153d28e6-5d08-4064-9c61-2dd05100b2d0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/153d28e6-5d08-4064-9c61-2dd05100b2d0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/153d28e6-5d08-4064-9c61-2dd05100b2d0/volumes/kubernetes.io~projected/kube-api-access-44vxk:{mountpoint:/var/lib/kubelet/pods/153d28e6-5d08-4064-9c61-2dd05100b2d0/volumes/kubernetes.io~projected/kube-api-access-44vxk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/153d28e6-5d08-4064-9c61-2dd05100b2d0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/153d28e6-5d08-4064-9c61-2dd05100b2d0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22fb35fa-b562-4179-b2c5-9d8e3976bf2d/volumes/kubernetes.io~projected/kube-api-access-f9t4x:{mountpoint:/var/lib/kubelet/pods/22fb35fa-b562-4179-b2c5-9d8e3976bf2d/volumes/kubernetes.io~projected/kube-api-access-f9t4x major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970/volumes/kubernetes.io~projected/kube-api-access-dzcb8:{mountpoint:/var/lib/kubelet/pods/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970/volumes/kubernetes.io~projected/kube-api-access-dzcb8 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b06086c-7464-4be0-a0c6-7568cb1dcd30/volumes/kubernetes.io~projected/kube-api-access-xwcfj:{mountpoint:/var/lib/kubelet/pods/6b06086c-7464-4be0-a0c6-7568cb1dcd30/volumes/kubernetes.io~projected/kube-api-access-xwcfj major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b091fb4-183e-41e9-a408-84fd1ca8f184/volumes/kubernetes.io~projected/kube-api-access-jqvn9:{mountpoint:/var/lib/kubelet/pods/6b091fb4-183e-41e9-a408-84fd1ca8f184/volumes/kubernetes.io~projected/kube-api-access-jqvn9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76b201e3-7c0b-4810-9baa-f6e054cf0826/volumes/kubernetes.io~projected/kube-api-access-nsncz:{mountpoint:/var/lib/kubelet/pods/76b201e3-7c0b-4810-9baa-f6e054cf0826/volumes/kubernetes.io~projected/kube-api-access-nsncz major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf/volumes/kubernetes.io~projected/kube-api-access-94d26:{mountpoint:/var/lib/kubelet/pods/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf/volumes/kubernetes.io~projected/kube-api-access-94d26 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f8a5faf-6b8f-4340-8b4c-945f8d67e008/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7f8a5faf-6b8f-4340-8b4c-945f8d67e008/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f8a5faf-6b8f-4340-8b4c-945f8d67e008/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7f8a5faf-6b8f-4340-8b4c-945f8d67e008/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f8a5faf-6b8f-4340-8b4c-945f8d67e008/volumes/kubernetes.io~projected/kube-api-access-2fsqb:{mountpoint:/var/lib/kubelet/pods/7f8a5faf-6b8f-4340-8b4c-945f8d67e008/volumes/kubernetes.io~projected/kube-api-access-2fsqb major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93c86199-dd69-41a7-84c6-fd3725cfa9f1/volumes/kubernetes.io~projected/kube-api-access-kwfvn:{mountpoint:/var/lib/kubelet/pods/93c86199-dd69-41a7-84c6-fd3725cfa9f1/volumes/kubernetes.io~projected/kube-api-access-kwfvn major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0475429-6780-4dd9-8185-199a83ce41f5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a0475429-6780-4dd9-8185-199a83ce41f5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc56646d-7901-4a14-a4af-46d2582ccc5c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cc56646d-7901-4a14-a4af-46d2582ccc5c/volumes/kubernetes.io~secret/agent-certs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d/volumes/kubernetes.io~projected/kube-api-access-xgzdr:{mountpoint:/var/lib/kubelet/pods/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d/volumes/kubernetes.io~projected/kube-api-access-xgzdr major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c61e8cdb37a7051e4032b57ef012c3985819ecaaa9fed5d3c8ecd28fe387b31a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1124e4c610571fcd34b87d1a3385b208a1b6dc35ada6f7c4f79ec152463365e3/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/45a6f2f0285b9279fdedca660015ab602baaf46e23680f6df4f312d99edaa8a4/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/containers/storage/overlay/74779eb48ead1d0bb85fa656ee196c4cc8fb8aef63b1fafc78f1b7c3586b8609/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/acf79ef4cc106fddd89edca9ff45865d724a753008e99b43e6b4a1f6b1e4f2b7/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/193a8b595d126ea1e7434d0013c87ef549b5c2e16e413118d3437ee649a550ff/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d13dc09e5d87335272a236ffa3cbe244fc51d27f76731b5465fe762eba6659fe/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c19579130a789b1b85878873a4d5e93ed616a94dee460ed8ad9e331f2b02910b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a8f2361982d7decc43fa585b881d37750055de8c5b2a92de376ef582776613dc/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b1ca8f487cd02036922639e59cc98b8a7154520fa34398875c37f63485fae182/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/171b7404f29ed9c89ecdb0dc85e8e8a420edef64b9b569275ec06503da69fccd/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/d3b6c301ba92e3d18d77b727888a07dd160c42674d4032cef22e04ded77bbb20/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/fba25428cf1dd702b891503757105f9eb84923b1f0dc1307e75ac62b6932aade/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6d6af089d332d276064d415472866cb002495e5de44e74db8e55c5f2572fbae8/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/aa43ebf08ecc80c42e5cdc8f1b8761f4db4226a066b8e5208b133ea32e2f03ab/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e68e64f88ed7fa1bda219db300cd41d455e74da60ce2964c5b9e771d64a15512/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/74c8ffb6c78bec1d5d024ce141145198701de3aab02963ceec3479160049c329/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/61ee91f009035c64c251372b5793238a2caa42055b39114971b892e742231ce6/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e39a7d2cd46da3da93fedeea9fa2d53c2e6d90c2c31f2e8ee9e57b362be72c81/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c39ad7d2f1c91e243fe8f7e3ea4ee512ca26571347f2d12c17633d203aebf258/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/cb82054e27e1c3b7ae84127f2695c406a258b0ab29b52b9f1d03c6b9c3be9269/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/f870714fca5595f88ae9dfc98d1a437dfb022920e17f4cb40d0659aa3491e810/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/e0869d966e16cf2c85d7544041ab7b79b76c2fc81bb12c90a0ebfc0362c031ef/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/24f7a46e1f66d862819151eef87379e007cf105132bbaa60d7058f8c653e1148/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/a8648d9c985a15c259623d3b95cb36914ed80f2ff2c083143ac8b956510d2752/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/1eea377a42378a94e529549faabae34621a161b9b8a666b4b90c80303e131429/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/38902d2f1c36dd59cd4e999c7ef3f2bf2e012b69538f562a8e95011399202a52/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/d46ef1f8e9500a6fa5f62aee7bb3659c7761cd24b397767354ae3341c5c34f35/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/de4f89e7b60cbba6d927955d6ebf1557de6df50ba2fba0b854f1137176dde9e3/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/0eb93e149d387bff617f28e3700c11368bd9bf59f0d5150bf5b1d87e7cc54fa3/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/b7c0b7829f8eed6412de4206c63f4fd473bb12e7af0b17d721333df50776fabb/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/f186526d7c5bad0ac42e9f1fcfffd100334829ee5c76a369de557a994e78a949/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/bbc1e720e6117f647faa6ac5466835435bc425e056ed7740e38f3eb95525840c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d81739c1d14eaf5347a1c658a1baa34cd009501b2796f855bea0bef726bf6c40/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/bf8477d7308f46f9f66d52b6b4417d507ff4bb2d90afc5fef9cee1138417705d/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.075324 4298 manager.go:217] Machine: {Timestamp:2026-11-03 22:07:23.074852871 +0000 UTC m=+0.055586515 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099385 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa93931d0687f87b29780675c8062 SystemUUID:ec2aa939-31d0-687f-87b2-9780675c8062 BootID:ba326697-dcb3-46f8-ac96-02c76373c5e7 Filesystems:[{Device:/var/lib/kubelet/pods/7f8a5faf-6b8f-4340-8b4c-945f8d67e008/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/6b091fb4-183e-41e9-a408-84fd1ca8f184/volumes/kubernetes.io~projected/kube-api-access-jqvn9 DeviceMajor:0 DeviceMinor:59 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b55817bda055bd396d0500ade254739abe9045140f0c9e239d6b02286d133066/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/188d93f62ce34515ada60b28aeec83adf435822944c8f12b7bcfedf56a1366ec/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d/volumes/kubernetes.io~projected/kube-api-access-xgzdr DeviceMajor:0 DeviceMinor:58 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/76b201e3-7c0b-4810-9baa-f6e054cf0826/volumes/kubernetes.io~projected/kube-api-access-nsncz DeviceMajor:0 DeviceMinor:64 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970/volumes/kubernetes.io~projected/kube-api-access-dzcb8 DeviceMajor:0 DeviceMinor:67 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/153d28e6-5d08-4064-9c61-2dd05100b2d0/volumes/kubernetes.io~projected/kube-api-access-44vxk DeviceMajor:0 DeviceMinor:60 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7f8a5faf-6b8f-4340-8b4c-945f8d67e008/volumes/kubernetes.io~projected/kube-api-access-2fsqb DeviceMajor:0 DeviceMinor:61 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2603ad2f8efbb8703e80500f47b3dc486eb54bff35395fcdfa0c9883009e8286/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-112 DeviceMajor:0 DeviceMinor:112 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc56646d-7901-4a14-a4af-46d2582ccc5c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:62 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93c86199-dd69-41a7-84c6-fd3725cfa9f1/volumes/kubernetes.io~projected/kube-api-access-kwfvn DeviceMajor:0 DeviceMinor:63 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10d481d4cd6bf34ce4eb7f990c6072a5493321b75cfc6bf48672cc526a9258fc/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/153d28e6-5d08-4064-9c61-2dd05100b2d0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ce713afc7eaf63a516c415eff23041ed2464eefebb24c6ca371432da39f7410/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b84b3417900acb306503f87c22b359da1caaab609ba89f8472ad54484c11278/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b06086c-7464-4be0-a0c6-7568cb1dcd30/volumes/kubernetes.io~projected/kube-api-access-xwcfj DeviceMajor:0 DeviceMinor:45 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/7f8a5faf-6b8f-4340-8b4c-945f8d67e008/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/22fb35fa-b562-4179-b2c5-9d8e3976bf2d/volumes/kubernetes.io~projected/kube-api-access-f9t4x DeviceMajor:0 DeviceMinor:66 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6094815b5ac2e90de4c5282bb5ccd036fb0758d0919f6ee44149a20a5f7bd31/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0475429-6780-4dd9-8185-199a83ce41f5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:197 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94005349a5bfe1a9a5137505223eefe7d3ea3d7ccec33d0cb5c8dddab9eaf14e/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85f69c9fc91998bfb5d5cdf395dfedfd5d780b67180868f2909ad106762380be/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/faa9d185fdd080afa6ebb5daae76e0ee73cdbc0f600922f6d4e75e94f18b3a87/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf/volumes/kubernetes.io~projected/kube-api-access-94d26 DeviceMajor:0 DeviceMinor:65 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/550f19f0fa67d96d43c1ab43cd2db377b82f2878eda402cf7a5d968cc7fcf04b/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/153d28e6-5d08-4064-9c61-2dd05100b2d0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bb5b0538a641eef684f65d2de9ed7ce71d94c8a15e307da2da9eba1c9139ab5/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:10d481d4cd6bf34 MacAddress:ca:40:27:59:d0:77 Speed:10000 Mtu:8901} {Name:188d93f62ce3451 MacAddress:66:06:49:47:dd:2b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d5:ed:97:33 Speed:0 Mtu:9001} {Name:br-int MacAddress:52:9b:fe:b7:7b:aa Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:d5:ed:97:33 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9a:24:c1:f1:32:02 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a2:de:50:99:9b:20 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.075554 4298 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.075603 4298 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.075977 4298 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.075998 4298 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-254.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076123 4298 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076131 4298 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076141 4298 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076158 4298 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076250 4298 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076413 4298 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076519 4298 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076555 4298 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:07:23.080027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076570 4298 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:07:23.080854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076581 4298 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:07:23.080854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076590 4298 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:07:23.080854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.076597 4298 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:07:23.080854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.077489 4298 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:07:23.080854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.077507 4298 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:07:23.088059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.088033 4298 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:07:23.089831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.089818 4298 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:07:23.089831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.089831 4298 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:07:23.089945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.089936 4298 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 22:07:23.090139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090128 4298 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:07:23.090174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090157 4298 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:07:23.090251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090242 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:07:23.090281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090257 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:07:23.090281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090263 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:07:23.090281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090269 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:07:23.090281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090275 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:07:23.090386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090283 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:07:23.090386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090289 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:07:23.090386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090296 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:07:23.090386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090305 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:07:23.090386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090312 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:07:23.090386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090323 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:07:23.090386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090334 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:07:23.090386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090354 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:07:23.090386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.090360 4298 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:07:23.093683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.093670 4298 server.go:1263] "Started kubelet" Nov 03 22:07:23.093964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.093755 4298 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:07:23.094220 ip-10-0-139-254 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:07:23.095080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.093797 4298 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:07:23.095080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.094550 4298 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:07:23.095080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.094908 4298 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:07:23.095258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.095245 4298 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:07:23.096758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.096740 4298 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:07:23.097560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.096834 4298 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:07:23.097560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.096862 4298 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:07:23.097560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.096985 4298 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:07:23.097560 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:23.096858 4298 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:07:23.098017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098000 4298 factory.go:55] Registering systemd factory Nov 03 22:07:23.098218 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098023 4298 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:07:23.098325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098280 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22fb35fa-b562-4179-b2c5-9d8e3976bf2d" volumeName="kubernetes.io/configmap/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-iptables-alerter-script" seLinuxMountContext="" Nov 03 22:07:23.098412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098324 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b091fb4-183e-41e9-a408-84fd1ca8f184" volumeName="kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-cni-binary-copy" seLinuxMountContext="" Nov 03 22:07:23.098412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098349 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b091fb4-183e-41e9-a408-84fd1ca8f184" volumeName="kubernetes.io/projected/6b091fb4-183e-41e9-a408-84fd1ca8f184-kube-api-access-jqvn9" seLinuxMountContext="" Nov 03 22:07:23.098412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098364 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b201e3-7c0b-4810-9baa-f6e054cf0826" volumeName="kubernetes.io/projected/76b201e3-7c0b-4810-9baa-f6e054cf0826-kube-api-access-nsncz" seLinuxMountContext="" Nov 03 22:07:23.098412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098377 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93c86199-dd69-41a7-84c6-fd3725cfa9f1" volumeName="kubernetes.io/projected/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kube-api-access-kwfvn" seLinuxMountContext="" Nov 03 22:07:23.098412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098399 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc56646d-7901-4a14-a4af-46d2582ccc5c" volumeName="kubernetes.io/secret/cc56646d-7901-4a14-a4af-46d2582ccc5c-agent-certs" seLinuxMountContext="" Nov 03 22:07:23.098716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098420 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d" volumeName="kubernetes.io/projected/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-kube-api-access-xgzdr" seLinuxMountContext="" Nov 03 22:07:23.098716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098432 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153d28e6-5d08-4064-9c61-2dd05100b2d0" volumeName="kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-env-overrides" seLinuxMountContext="" Nov 03 22:07:23.098716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098629 4298 factory.go:153] Registering CRI-O factory Nov 03 22:07:23.098716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098661 4298 factory.go:223] Registration of the crio container factory successfully Nov 03 22:07:23.098716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098717 4298 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:07:23.098945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098738 4298 factory.go:103] Registering Raw factory Nov 03 22:07:23.098945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098753 4298 manager.go:1199] Started watching for new ooms in manager Nov 03 22:07:23.099026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.098977 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153d28e6-5d08-4064-9c61-2dd05100b2d0" volumeName="kubernetes.io/secret/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 22:07:23.099026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099005 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22fb35fa-b562-4179-b2c5-9d8e3976bf2d" volumeName="kubernetes.io/projected/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-kube-api-access-f9t4x" seLinuxMountContext="" Nov 03 22:07:23.099026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099020 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b06086c-7464-4be0-a0c6-7568cb1dcd30" volumeName="kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj" seLinuxMountContext="" Nov 03 22:07:23.099110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099044 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b091fb4-183e-41e9-a408-84fd1ca8f184" volumeName="kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-daemon-config" seLinuxMountContext="" Nov 03 22:07:23.099110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099057 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f8a5faf-6b8f-4340-8b4c-945f8d67e008" volumeName="kubernetes.io/projected/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-kube-api-access-2fsqb" seLinuxMountContext="" Nov 03 22:07:23.099110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099068 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0475429-6780-4dd9-8185-199a83ce41f5" volumeName="kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret" seLinuxMountContext="" Nov 03 22:07:23.099110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099087 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc56646d-7901-4a14-a4af-46d2582ccc5c" volumeName="kubernetes.io/configmap/cc56646d-7901-4a14-a4af-46d2582ccc5c-konnectivity-ca" seLinuxMountContext="" Nov 03 22:07:23.099110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099100 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153d28e6-5d08-4064-9c61-2dd05100b2d0" volumeName="kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-script-lib" seLinuxMountContext="" Nov 03 22:07:23.099264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099115 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f8a5faf-6b8f-4340-8b4c-945f8d67e008" volumeName="kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-tuned" seLinuxMountContext="" Nov 03 22:07:23.099264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099129 4298 manager.go:319] Starting recovery of all containers Nov 03 22:07:23.099470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099386 4298 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:07:23.099594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099547 4298 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:01:42 +0000 UTC" deadline="2028-08-14 07:15:50.656546788 +0000 UTC" Nov 03 22:07:23.099594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099595 4298 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15585h8m27.556955073s" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.099128 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d" volumeName="kubernetes.io/empty-dir/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-tmp-dir" seLinuxMountContext="" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.101330 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153d28e6-5d08-4064-9c61-2dd05100b2d0" volumeName="kubernetes.io/projected/153d28e6-5d08-4064-9c61-2dd05100b2d0-kube-api-access-44vxk" seLinuxMountContext="" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.101353 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" volumeName="kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-binary-copy" seLinuxMountContext="" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.101366 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" volumeName="kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.101379 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" volumeName="kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.101391 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" volumeName="kubernetes.io/projected/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-kube-api-access-dzcb8" seLinuxMountContext="" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.101403 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b201e3-7c0b-4810-9baa-f6e054cf0826" volumeName="kubernetes.io/configmap/76b201e3-7c0b-4810-9baa-f6e054cf0826-serviceca" seLinuxMountContext="" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.101416 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77d5bb8c-19f0-4d74-b146-ae9d69aebcdf" volumeName="kubernetes.io/projected/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-kube-api-access-94d26" seLinuxMountContext="" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.101427 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f8a5faf-6b8f-4340-8b4c-945f8d67e008" volumeName="kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-tmp" seLinuxMountContext="" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.101438 4298 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153d28e6-5d08-4064-9c61-2dd05100b2d0" volumeName="kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-config" seLinuxMountContext="" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.101449 4298 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:07:23.101810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.101455 4298 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:07:23.125484 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.125445 4298 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:07:23.127158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.127069 4298 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:07:23.127158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.127096 4298 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:07:23.127158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.127120 4298 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:07:23.127349 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:23.127164 4298 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 22:07:23.137554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.137525 4298 generic.go:356] "Generic (PLEG): container finished" podID="00032b9e4863070a848e5aad12046474" containerID="5ab761798d231fb90e25afcb65f7f11d7028007cd372627687c7b82ba6149211" exitCode=0 Nov 03 22:07:23.144353 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.144323 4298 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="3be48978eb05ab493fc25644ed368bb31f7d896a6731e6993edd85dab8e72f93" exitCode=0 Nov 03 22:07:23.144353 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.144351 4298 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="881c2e2243b5d7e0a262ad4bedca718cdb9430913e1783d029536bb403d9ebfa" exitCode=0 Nov 03 22:07:23.144353 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.144359 4298 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="99dbb9b42ce4464fdf078eb2116b066256634d235b859f8892906ee2733410f0" exitCode=0 Nov 03 22:07:23.144571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.144366 4298 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="88414a89a81722716e13394ddc72bab71f1726b8135666b41833b7e82eec1417" exitCode=0 Nov 03 22:07:23.144571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.144373 4298 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="f0d9aa308458cb37754acaa469bd713b19aad89298e7cbeb2f03161ccfabc0e3" exitCode=0 Nov 03 22:07:23.144571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.144380 4298 generic.go:356] "Generic (PLEG): container finished" podID="355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970" containerID="ee1d7ca2921e7ed19a7da63561ff9584710ce6a08cc42b590ace5b87d69cea9a" exitCode=0 Nov 03 22:07:23.160823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.160793 4298 manager.go:324] Recovery completed Nov 03 22:07:23.172254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.172235 4298 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:07:23.173504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.173489 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:07:23.173555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.173516 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:07:23.173555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.173527 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:07:23.174734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.174720 4298 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:07:23.174734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.174733 4298 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:07:23.174813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.174749 4298 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:07:23.174813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.174733 4298 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:07:23.174877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.174864 4298 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 22:07:23.174910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.174878 4298 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 22:07:23.174910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.174897 4298 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 22:07:23.174910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.174904 4298 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 22:07:23.174995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.174912 4298 policy_none.go:50] "Start" Nov 03 22:07:23.174995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.174922 4298 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:07:23.174995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.174932 4298 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:07:23.175091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.175081 4298 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 22:07:23.175120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.175092 4298 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 22:07:23.175120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.175098 4298 policy_none.go:44] "Start" Nov 03 22:07:23.175743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.175726 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:07:23.175802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.175761 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:07:23.175802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.175776 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:07:23.179067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.179043 4298 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:07:23.181695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.181683 4298 manager.go:341] "Starting Device Plugin manager" Nov 03 22:07:23.181738 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:23.181710 4298 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:07:23.181738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.181720 4298 server.go:87] "Starting device plugin registration server" Nov 03 22:07:23.181987 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.181973 4298 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:07:23.182042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.181984 4298 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:07:23.182088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.182074 4298 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:07:23.182190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.182159 4298 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:07:23.182262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.182197 4298 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:07:23.182262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.182204 4298 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:07:23.182351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.182256 4298 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:07:23.182187593Z","UUID":"1f2e1a60-7b0a-4589-ab72-ce85fb2e61f8","Handler":null,"Name":"","Endpoint":""} Nov 03 22:07:23.183364 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:23.183341 4298 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:07:23.183444 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:23.183386 4298 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" Nov 03 22:07:23.183690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.183672 4298 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:07:23.183774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.183697 4298 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:07:23.227768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.227294 4298 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal"] Nov 03 22:07:23.227938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.227865 4298 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:07:23.229081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.229063 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:07:23.229154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.229093 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:07:23.229154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.229104 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:07:23.229345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.229334 4298 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:07:23.229515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.229500 4298 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:07:23.230347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230330 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:07:23.230403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230363 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:07:23.230403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230373 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:07:23.230464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230330 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:07:23.230464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230442 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:07:23.230464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230461 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:07:23.230556 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:23.230513 4298 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.230687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230635 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" event={"ID":"00032b9e4863070a848e5aad12046474","Type":"ContainerStarted","Data":"b3a3296e32c3c46ad8a3b663d41c4c4fdbb6f1402cfff07ae1153d21edce4c16"} Nov 03 22:07:23.230718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230695 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" event={"ID":"00032b9e4863070a848e5aad12046474","Type":"ContainerDied","Data":"5ab761798d231fb90e25afcb65f7f11d7028007cd372627687c7b82ba6149211"} Nov 03 22:07:23.230718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230712 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" event={"ID":"00032b9e4863070a848e5aad12046474","Type":"ContainerStarted","Data":"8ce713afc7eaf63a516c415eff23041ed2464eefebb24c6ca371432da39f7410"} Nov 03 22:07:23.230718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230715 4298 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:07:23.230799 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230777 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" event={"ID":"815f1999fc2bcba4fa27ffca21498e4f","Type":"ContainerStarted","Data":"a370575ad259627e29d20536603aab751df0953decd8750649b88ffe21bafcb2"} Nov 03 22:07:23.230830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.230796 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" event={"ID":"815f1999fc2bcba4fa27ffca21498e4f","Type":"ContainerStarted","Data":"85f69c9fc91998bfb5d5cdf395dfedfd5d780b67180868f2909ad106762380be"} Nov 03 22:07:23.231558 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.231543 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:07:23.231618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.231566 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:07:23.231618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.231578 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:07:23.231723 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:23.231711 4298 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-254.us-west-2.compute.internal\" not found" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.282528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.282492 4298 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:07:23.283658 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.283628 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:07:23.283753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.283674 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:07:23.283753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.283685 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:07:23.283753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.283704 4298 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.294844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.294825 4298 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:07:23.345291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.345222 4298 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:07:23.345845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.345827 4298 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 22:07:23.347372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.347354 4298 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:07:23.396329 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.396291 4298 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:07:23.397472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.397452 4298 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.446922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.446889 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.446922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.446924 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.447101 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.446941 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/815f1999fc2bcba4fa27ffca21498e4f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"815f1999fc2bcba4fa27ffca21498e4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.449018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.448995 4298 nodeinfomanager.go:471] existing CSINode "ip-10-0-139-254.us-west-2.compute.internal" is owned by different node (oldNodeID="6f4e212f-eeca-4ce1-bbb1-8557306f0877", newNodeID=""), cleaning up... Nov 03 22:07:23.449188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.449173 4298 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.449244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.449236 4298 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.449824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.449810 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeNotReady" Nov 03 22:07:23.449901 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.449843 4298 setters.go:546] "Node became not ready" node="ip-10-0-139-254.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T22:07:23Z","lastTransitionTime":"2026-11-03T22:07:23Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 22:07:23.456327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.456301 4298 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:07:23.456411 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:23.456353 4298 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.456411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.456368 4298 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.515504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.515474 4298 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:07:23.515631 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:23.515518 4298 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.547332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.547293 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.547332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.547336 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.547517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.547393 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/815f1999fc2bcba4fa27ffca21498e4f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"815f1999fc2bcba4fa27ffca21498e4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.547517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.547414 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.547517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.547439 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/815f1999fc2bcba4fa27ffca21498e4f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"815f1999fc2bcba4fa27ffca21498e4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.547517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.547471 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00032b9e4863070a848e5aad12046474-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" (UID: \"00032b9e4863070a848e5aad12046474\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:23.716586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.716558 4298 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-254.us-west-2.compute.internal" event="NodeReady" Nov 03 22:07:23.716766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:23.716667 4298 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 22:07:24.077760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.077679 4298 apiserver.go:52] "Watching apiserver" Nov 03 22:07:24.166144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.166102 4298 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:24.166373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.166342 4298 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:24.182105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.182069 4298 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:07:24.182439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.182415 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-f8hqj","openshift-dns/node-resolver-6q9pz","openshift-network-diagnostics/network-check-target-bljnw","openshift-ovn-kubernetes/ovnkube-node-5gw2b","kube-system/global-pull-secret-syncer-7zmnf","kube-system/konnectivity-agent-b94tl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh","openshift-cluster-node-tuning-operator/tuned-h5mwd","openshift-image-registry/node-ca-k98wl","openshift-multus/multus-r2k9j","openshift-multus/network-metrics-daemon-4m4qq","openshift-network-operator/iptables-alerter-lr6gj","kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal"] Nov 03 22:07:24.185977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.185944 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:24.198210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.198191 4298 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:07:24.227195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.227172 4298 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:07:24.227324 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:24.227224 4298 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:24.227566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.227551 4298 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:07:24.227600 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:24.227588 4298 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-254.us-west-2.compute.internal" Nov 03 22:07:24.239428 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239396 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:07:24.239519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239484 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:07:24.239519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239496 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:07:24.239625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239532 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:07:24.239688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239656 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:07:24.239688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239671 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:07:24.239777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239727 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:07:24.239777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239747 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:07:24.239777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239773 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:07:24.239916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239813 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:07:24.239916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239866 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ng2g2\"" Nov 03 22:07:24.239916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239908 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z5g9w\"" Nov 03 22:07:24.239916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239915 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:07:24.240094 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239967 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:07:24.240094 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.239985 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:07:24.240094 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240001 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:07:24.240094 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240060 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:24.240382 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240364 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j54w4\"" Nov 03 22:07:24.240500 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240484 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8pftz\"" Nov 03 22:07:24.240561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240522 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:07:24.240561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240527 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:07:24.240561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240428 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:07:24.240561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240437 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:24.240561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240546 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:07:24.240561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240438 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ckrtj\"" Nov 03 22:07:24.240561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240476 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240476 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240482 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240481 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240483 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g6ff6\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240614 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240502 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m27qh\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240524 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kxpq\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240396 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bkplg\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240669 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dngqk\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240437 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240548 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240455 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dgskz\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240378 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240712 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:07:24.240803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.240726 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:07:24.250006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.249988 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.250054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250013 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-device-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.250054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250034 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-sys-fs\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.250106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250077 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-ovn\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.250144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250110 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.250144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250136 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-socket-dir-parent\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.250205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250172 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:24.250205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250201 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-tmp-dir\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:07:24.250262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250235 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-netd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.250309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250295 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-tmp-dir\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:07:24.250355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250345 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-tmp\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.250414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250400 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-socket-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.250464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250451 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-tmp\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.250494 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250488 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-etc-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.250525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250515 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-node-log\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.250556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250531 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.250584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250559 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysconfig\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.250614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250584 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-sys\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.250614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250606 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-bin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.250732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250625 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76b201e3-7c0b-4810-9baa-f6e054cf0826-host\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:07:24.250732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250667 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94d26\" (UniqueName: \"kubernetes.io/projected/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-kube-api-access-94d26\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:24.250732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250683 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-hosts-file\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:07:24.250732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250698 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-kubelet\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.250732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250714 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-binary-copy\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.250923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.250732 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:24.251102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.251079 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0475429-6780-4dd9-8185-199a83ce41f5-original-pull-secret\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:24.251605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.251079 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-binary-copy\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.252078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.251839 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfvn\" (UniqueName: \"kubernetes.io/projected/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kube-api-access-kwfvn\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.252078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.251871 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:24.252078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.251880 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5mdhj\"" Nov 03 22:07:24.252078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.251895 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-host-slash\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:07:24.252078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.251920 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-slash\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.252078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.251980 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vxk\" (UniqueName: \"kubernetes.io/projected/153d28e6-5d08-4064-9c61-2dd05100b2d0-kube-api-access-44vxk\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.252078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252064 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-tuned\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.252455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252198 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-tuned\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.252455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252245 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fsqb\" (UniqueName: \"kubernetes.io/projected/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-kube-api-access-2fsqb\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.252455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252305 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-etc-kubernetes\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.252455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252350 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cnibin\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.252455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252386 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/76b201e3-7c0b-4810-9baa-f6e054cf0826-serviceca\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:07:24.252455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252413 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-config\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.252455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252450 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc56646d-7901-4a14-a4af-46d2582ccc5c-konnectivity-ca\") pod \"konnectivity-agent-b94tl\" (UID: \"cc56646d-7901-4a14-a4af-46d2582ccc5c\") " pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:07:24.252754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252483 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-kubernetes\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.252754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252523 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-lib-modules\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.252754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252588 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-conf-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.252754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252618 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9t4x\" (UniqueName: \"kubernetes.io/projected/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-kube-api-access-f9t4x\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:07:24.252754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252715 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-system-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.252956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.252912 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc56646d-7901-4a14-a4af-46d2582ccc5c-konnectivity-ca\") pod \"konnectivity-agent-b94tl\" (UID: \"cc56646d-7901-4a14-a4af-46d2582ccc5c\") " pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:07:24.253153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253124 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-config\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.253213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253177 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-multus\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.253213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253208 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-daemon-config\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.253330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253242 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-multus-certs\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.253330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253282 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvn9\" (UniqueName: \"kubernetes.io/projected/6b091fb4-183e-41e9-a408-84fd1ca8f184-kube-api-access-jqvn9\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.253330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253314 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcb8\" (UniqueName: \"kubernetes.io/projected/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-kube-api-access-dzcb8\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.253490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253342 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-systemd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.253490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253370 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc56646d-7901-4a14-a4af-46d2582ccc5c-agent-certs\") pod \"konnectivity-agent-b94tl\" (UID: \"cc56646d-7901-4a14-a4af-46d2582ccc5c\") " pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:07:24.253490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253401 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.253490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253432 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-dbus\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:24.253490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253466 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-iptables-alerter-script\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:07:24.253816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253501 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgzdr\" (UniqueName: \"kubernetes.io/projected/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-kube-api-access-xgzdr\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:07:24.253816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253529 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-k8s-cni-cncf-io\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.253816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253562 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.253816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253593 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.253816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253623 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovn-node-metrics-cert\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.253816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253675 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.253816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253700 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-run\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.253816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253735 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-cni-binary-copy\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.253816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253766 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-kubelet-config\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:24.253816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253785 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/76b201e3-7c0b-4810-9baa-f6e054cf0826-serviceca\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:07:24.253816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253796 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-registration-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.254449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253826 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-kubelet\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.254449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253850 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-bin\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.254449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253920 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-systemd\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.254449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253961 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-cnibin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.254449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253973 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc56646d-7901-4a14-a4af-46d2582ccc5c-agent-certs\") pod \"konnectivity-agent-b94tl\" (UID: \"cc56646d-7901-4a14-a4af-46d2582ccc5c\") " pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:07:24.254449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.253993 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.254712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.254621 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-daemon-config\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.255162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.254875 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovn-node-metrics-cert\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.255162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255060 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-iptables-alerter-script\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:07:24.255162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255084 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.255162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255121 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsncz\" (UniqueName: \"kubernetes.io/projected/76b201e3-7c0b-4810-9baa-f6e054cf0826-kube-api-access-nsncz\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:07:24.255162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255128 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.255415 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255187 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-netns\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.255415 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255237 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-var-lib-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.255415 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255384 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b091fb4-183e-41e9-a408-84fd1ca8f184-cni-binary-copy\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.255589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255389 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.255589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255486 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-log-socket\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.255589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255570 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-script-lib\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.255963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255942 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-var-lib-kubelet\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.256031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.255985 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-conf\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.256089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.256031 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcfj\" (UniqueName: \"kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj\") pod \"network-check-target-bljnw\" (UID: \"6b06086c-7464-4be0-a0c6-7568cb1dcd30\") " pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:24.256089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.256060 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-systemd-units\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.256089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.256083 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-modprobe-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.256234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.256102 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.256234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.256165 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-netns\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.256234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.256174 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-ovnkube-script-lib\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.256234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.256205 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-hostroot\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.256385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.256255 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-system-cni-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.256385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.256286 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-env-overrides\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.256385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.256309 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-host\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.256465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.256444 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/153d28e6-5d08-4064-9c61-2dd05100b2d0-env-overrides\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.306709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.306673 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94d26\" (UniqueName: \"kubernetes.io/projected/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-kube-api-access-94d26\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:24.308237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.308212 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfvn\" (UniqueName: \"kubernetes.io/projected/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kube-api-access-kwfvn\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.308470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.308454 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vxk\" (UniqueName: \"kubernetes.io/projected/153d28e6-5d08-4064-9c61-2dd05100b2d0-kube-api-access-44vxk\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.310341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.310325 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fsqb\" (UniqueName: \"kubernetes.io/projected/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-kube-api-access-2fsqb\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.310378 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.310330 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9t4x\" (UniqueName: \"kubernetes.io/projected/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-kube-api-access-f9t4x\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:07:24.311204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.311183 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvn9\" (UniqueName: \"kubernetes.io/projected/6b091fb4-183e-41e9-a408-84fd1ca8f184-kube-api-access-jqvn9\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.311327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.311312 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgzdr\" (UniqueName: \"kubernetes.io/projected/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-kube-api-access-xgzdr\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:07:24.311742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.311727 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzcb8\" (UniqueName: \"kubernetes.io/projected/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-kube-api-access-dzcb8\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.311882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.311867 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwcfj\" (UniqueName: \"kubernetes.io/projected/6b06086c-7464-4be0-a0c6-7568cb1dcd30-kube-api-access-xwcfj\") pod \"network-check-target-bljnw\" (UID: \"6b06086c-7464-4be0-a0c6-7568cb1dcd30\") " pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:24.312081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.312064 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsncz\" (UniqueName: \"kubernetes.io/projected/76b201e3-7c0b-4810-9baa-f6e054cf0826-kube-api-access-nsncz\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:07:24.357052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.356989 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-etc-kubernetes\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357020 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cnibin\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.357052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357037 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-kubernetes\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357051 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-lib-modules\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357067 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-conf-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357082 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-system-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357097 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-multus\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357103 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-etc-kubernetes\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357103 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-cnibin\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357141 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-kubernetes\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357146 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-conf-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357147 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-system-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357193 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-multus-certs\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357207 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-lib-modules\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357223 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-multus\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357236 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-multus-certs\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357224 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-systemd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357260 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-systemd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357283 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357307 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-dbus\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357327 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-k8s-cni-cncf-io\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357343 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357358 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357370 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357376 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-run\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357406 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-k8s-cni-cncf-io\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357418 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357423 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-kubelet-config\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357436 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-run\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357436 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-dbus\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357452 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-registration-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357477 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-kubelet\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357481 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357483 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-registration-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357481 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0475429-6780-4dd9-8185-199a83ce41f5-kubelet-config\") pod \"global-pull-secret-syncer-7zmnf\" (UID: \"a0475429-6780-4dd9-8185-199a83ce41f5\") " pod="kube-system/global-pull-secret-syncer-7zmnf" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357499 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-bin\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357523 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-kubelet\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357546 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-systemd\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357560 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-cnibin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357591 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-cnibin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357590 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-systemd\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357596 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-netns\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357609 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-bin\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357624 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-var-lib-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357656 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357671 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-log-socket\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357685 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-netns\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357689 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-var-lib-kubelet\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357706 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357719 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-var-lib-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357757 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-conf\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357785 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-systemd-units\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357801 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-var-lib-kubelet\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357801 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-log-socket\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357828 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-modprobe-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357827 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-systemd-units\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357853 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.357853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357854 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysctl-conf\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357893 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-cni-dir\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357919 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-modprobe-d\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357915 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-netns\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357957 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-hostroot\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357974 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-system-cni-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.357998 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-run-netns\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358006 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970-system-cni-dir\") pod \"multus-additional-cni-plugins-f8hqj\" (UID: \"355e7f6f-eb2e-47a9-b4f3-0f7cfd5cb970\") " pod="openshift-multus/multus-additional-cni-plugins-f8hqj" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358020 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-hostroot\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358037 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-host\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358059 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358080 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-host\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358105 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-device-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358119 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358145 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-device-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358149 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-sys-fs\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358121 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-sys-fs\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358175 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-ovn\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358193 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358208 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-socket-dir-parent\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358223 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358238 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-netd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358242 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-run-ovn\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358259 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358278 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-multus-socket-dir-parent\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358280 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-cni-netd\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358263 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-socket-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358316 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-etc-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358351 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-etc-openvswitch\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358366 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93c86199-dd69-41a7-84c6-fd3725cfa9f1-socket-dir\") pod \"aws-ebs-csi-driver-node-x8qfh\" (UID: \"93c86199-dd69-41a7-84c6-fd3725cfa9f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8qfh" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358368 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-node-log\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358390 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-node-log\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358395 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358415 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358420 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysconfig\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358436 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-sys\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358441 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-etc-sysconfig\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358450 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-bin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358464 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76b201e3-7c0b-4810-9baa-f6e054cf0826-host\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358466 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f8a5faf-6b8f-4340-8b4c-945f8d67e008-sys\") pod \"tuned-h5mwd\" (UID: \"7f8a5faf-6b8f-4340-8b4c-945f8d67e008\") " pod="openshift-cluster-node-tuning-operator/tuned-h5mwd" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358479 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-hosts-file\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358490 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-cni-bin\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358500 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-kubelet\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358518 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d-hosts-file\") pod \"node-resolver-6q9pz\" (UID: \"d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d\") " pod="openshift-dns/node-resolver-6q9pz" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358530 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76b201e3-7c0b-4810-9baa-f6e054cf0826-host\") pod \"node-ca-k98wl\" (UID: \"76b201e3-7c0b-4810-9baa-f6e054cf0826\") " pod="openshift-image-registry/node-ca-k98wl" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358538 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b091fb4-183e-41e9-a408-84fd1ca8f184-host-var-lib-kubelet\") pod \"multus-r2k9j\" (UID: \"6b091fb4-183e-41e9-a408-84fd1ca8f184\") " pod="openshift-multus/multus-r2k9j" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358541 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-host-slash\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358520 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22fb35fa-b562-4179-b2c5-9d8e3976bf2d-host-slash\") pod \"iptables-alerter-lr6gj\" (UID: \"22fb35fa-b562-4179-b2c5-9d8e3976bf2d\") " pod="openshift-network-operator/iptables-alerter-lr6gj" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358570 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-slash\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358613 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/153d28e6-5d08-4064-9c61-2dd05100b2d0-host-slash\") pod \"ovnkube-node-5gw2b\" (UID: \"153d28e6-5d08-4064-9c61-2dd05100b2d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:24.359529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.358632 4298 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:07:24.360842 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.360595 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d5bb8c-19f0-4d74-b146-ae9d69aebcdf-metrics-certs\") pod \"network-metrics-daemon-4m4qq\" (UID: \"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf\") " pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:24.487098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.487065 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m4qq" Nov 03 22:07:24.560040 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.560008 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:24.849709 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:24.849638 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d5bb8c_19f0_4d74_b146_ae9d69aebcdf.slice/crio-122577a46bc3cf47b0855db642e8c9149de7726d22e14241f9dd7c4b9c0bded7 WatchSource:0}: Error finding container 122577a46bc3cf47b0855db642e8c9149de7726d22e14241f9dd7c4b9c0bded7: Status 404 returned error can't find the container with id 122577a46bc3cf47b0855db642e8c9149de7726d22e14241f9dd7c4b9c0bded7 Nov 03 22:07:24.850596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.850568 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4m4qq"] Nov 03 22:07:24.851810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:24.851793 4298 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:07:25.168622 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:25.168535 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4m4qq" event={"ID":"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf","Type":"ContainerStarted","Data":"122577a46bc3cf47b0855db642e8c9149de7726d22e14241f9dd7c4b9c0bded7"} Nov 03 22:07:26.491949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:26.491923 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:26.881900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:26.881819 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:27.064512 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.064474 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:27.175476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.175381 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4m4qq" event={"ID":"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf","Type":"ContainerStarted","Data":"5620cea58fa43e320caabb7079a4330dde27485dd64e86fbb841d922ff42cf73"} Nov 03 22:07:27.175476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.175436 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4m4qq" event={"ID":"77d5bb8c-19f0-4d74-b146-ae9d69aebcdf","Type":"ContainerStarted","Data":"976428a246438527c103ddd13e5975924077752f7c33a92be37b8d296b1d9fd4"} Nov 03 22:07:27.374517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.374482 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-mj8hx"] Nov 03 22:07:27.377402 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.377379 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mj8hx" Nov 03 22:07:27.387632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.387607 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g74dr"] Nov 03 22:07:27.390246 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.390229 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g74dr" Nov 03 22:07:27.391096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.391078 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-56cpk"] Nov 03 22:07:27.393840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.393823 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.425958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.425879 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xgkft"] Nov 03 22:07:27.428722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.428698 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.435079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.435058 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 22:07:27.435201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.435126 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qbrgx\"" Nov 03 22:07:27.435201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.435166 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 22:07:27.450287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.447239 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:07:27.450287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.447360 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:07:27.476033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.475949 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d57db45fb-zszqm"] Nov 03 22:07:27.477465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.477436 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z86ph\" (UniqueName: \"kubernetes.io/projected/ad85509c-9e79-4bdc-a069-235d4af67c62-kube-api-access-z86ph\") pod \"downloads-6dd789c8b7-mj8hx\" (UID: \"ad85509c-9e79-4bdc-a069-235d4af67c62\") " pod="openshift-console/downloads-6dd789c8b7-mj8hx" Nov 03 22:07:27.477579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.477474 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chqcd\" (UniqueName: \"kubernetes.io/projected/d9504538-9906-445e-960c-5c7d9e0e7598-kube-api-access-chqcd\") pod \"ingress-canary-g74dr\" (UID: \"d9504538-9906-445e-960c-5c7d9e0e7598\") " pod="openshift-ingress-canary/ingress-canary-g74dr" Nov 03 22:07:27.477579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.477509 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d2dbde5-e102-48ee-a803-c14a844aaa9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.477668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.477633 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c545\" (UniqueName: \"kubernetes.io/projected/9d2dbde5-e102-48ee-a803-c14a844aaa9d-kube-api-access-4c545\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.477700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.477684 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad85509c-9e79-4bdc-a069-235d4af67c62-tmp\") pod \"downloads-6dd789c8b7-mj8hx\" (UID: \"ad85509c-9e79-4bdc-a069-235d4af67c62\") " pod="openshift-console/downloads-6dd789c8b7-mj8hx" Nov 03 22:07:27.477749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.477731 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9504538-9906-445e-960c-5c7d9e0e7598-cert\") pod \"ingress-canary-g74dr\" (UID: \"d9504538-9906-445e-960c-5c7d9e0e7598\") " pod="openshift-ingress-canary/ingress-canary-g74dr" Nov 03 22:07:27.477786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.477760 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d2dbde5-e102-48ee-a803-c14a844aaa9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.477786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.477780 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d2dbde5-e102-48ee-a803-c14a844aaa9d-data-volume\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.477848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.477815 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d2dbde5-e102-48ee-a803-c14a844aaa9d-crio-socket\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.478876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.478855 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:07:27.478876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.478864 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:07:27.479028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.478932 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:07:27.479070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.479058 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.483590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.483538 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:07:27.496255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.496230 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b94tl" Nov 03 22:07:27.534425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.534390 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 22:07:27.534748 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.534729 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:07:27.534832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.534729 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gsv4s\"" Nov 03 22:07:27.534900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.534888 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fcb7n\"" Nov 03 22:07:27.534944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.534893 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 22:07:27.547935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.547916 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:07:27.547935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.547925 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbwbg\"" Nov 03 22:07:27.548041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.547924 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:07:27.548041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.547917 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2d6mp\"" Nov 03 22:07:27.548110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.548098 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 22:07:27.578223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578195 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chqcd\" (UniqueName: \"kubernetes.io/projected/d9504538-9906-445e-960c-5c7d9e0e7598-kube-api-access-chqcd\") pod \"ingress-canary-g74dr\" (UID: \"d9504538-9906-445e-960c-5c7d9e0e7598\") " pod="openshift-ingress-canary/ingress-canary-g74dr" Nov 03 22:07:27.578390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578234 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17697329-5597-4aff-a1fa-9c55b5a63e62-bound-sa-token\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.578390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578259 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17697329-5597-4aff-a1fa-9c55b5a63e62-installation-pull-secrets\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.578390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578280 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d2dbde5-e102-48ee-a803-c14a844aaa9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.578545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578394 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c545\" (UniqueName: \"kubernetes.io/projected/9d2dbde5-e102-48ee-a803-c14a844aaa9d-kube-api-access-4c545\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.578545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578444 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22lbt\" (UniqueName: \"kubernetes.io/projected/9f5ed133-b829-4cf8-accf-9fbe62d82a31-kube-api-access-22lbt\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.578671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578569 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad85509c-9e79-4bdc-a069-235d4af67c62-tmp\") pod \"downloads-6dd789c8b7-mj8hx\" (UID: \"ad85509c-9e79-4bdc-a069-235d4af67c62\") " pod="openshift-console/downloads-6dd789c8b7-mj8hx" Nov 03 22:07:27.578671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578590 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9504538-9906-445e-960c-5c7d9e0e7598-cert\") pod \"ingress-canary-g74dr\" (UID: \"d9504538-9906-445e-960c-5c7d9e0e7598\") " pod="openshift-ingress-canary/ingress-canary-g74dr" Nov 03 22:07:27.578671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578612 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d2dbde5-e102-48ee-a803-c14a844aaa9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.578671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578638 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d2dbde5-e102-48ee-a803-c14a844aaa9d-data-volume\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.578922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578762 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17697329-5597-4aff-a1fa-9c55b5a63e62-registry-tls\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.578922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578789 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d2dbde5-e102-48ee-a803-c14a844aaa9d-crio-socket\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.578922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578808 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f5ed133-b829-4cf8-accf-9fbe62d82a31-metrics-tls\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.578922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578831 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5ed133-b829-4cf8-accf-9fbe62d82a31-tmp-dir\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.578922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578847 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17697329-5597-4aff-a1fa-9c55b5a63e62-image-registry-private-configuration\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.578922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578865 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17697329-5597-4aff-a1fa-9c55b5a63e62-ca-trust-extracted\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.578922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578895 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17697329-5597-4aff-a1fa-9c55b5a63e62-trusted-ca\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.578922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578921 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb97h\" (UniqueName: \"kubernetes.io/projected/17697329-5597-4aff-a1fa-9c55b5a63e62-kube-api-access-xb97h\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.579311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578943 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17697329-5597-4aff-a1fa-9c55b5a63e62-registry-certificates\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.579311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578967 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z86ph\" (UniqueName: \"kubernetes.io/projected/ad85509c-9e79-4bdc-a069-235d4af67c62-kube-api-access-z86ph\") pod \"downloads-6dd789c8b7-mj8hx\" (UID: \"ad85509c-9e79-4bdc-a069-235d4af67c62\") " pod="openshift-console/downloads-6dd789c8b7-mj8hx" Nov 03 22:07:27.579311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578974 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d2dbde5-e102-48ee-a803-c14a844aaa9d-data-volume\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.579311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.578985 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5ed133-b829-4cf8-accf-9fbe62d82a31-config-volume\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.579311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.579062 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad85509c-9e79-4bdc-a069-235d4af67c62-tmp\") pod \"downloads-6dd789c8b7-mj8hx\" (UID: \"ad85509c-9e79-4bdc-a069-235d4af67c62\") " pod="openshift-console/downloads-6dd789c8b7-mj8hx" Nov 03 22:07:27.579311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.579104 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d2dbde5-e102-48ee-a803-c14a844aaa9d-crio-socket\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.579311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.579289 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d2dbde5-e102-48ee-a803-c14a844aaa9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.580733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.580713 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d2dbde5-e102-48ee-a803-c14a844aaa9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.580916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.580901 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9504538-9906-445e-960c-5c7d9e0e7598-cert\") pod \"ingress-canary-g74dr\" (UID: \"d9504538-9906-445e-960c-5c7d9e0e7598\") " pod="openshift-ingress-canary/ingress-canary-g74dr" Nov 03 22:07:27.585808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.585780 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 22:07:27.586745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.586729 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g74dr"] Nov 03 22:07:27.587298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.587282 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-56cpk"] Nov 03 22:07:27.587937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.587926 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d57db45fb-zszqm"] Nov 03 22:07:27.588485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.588465 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mj8hx"] Nov 03 22:07:27.613793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.613767 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xgkft"] Nov 03 22:07:27.637623 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.637591 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c545\" (UniqueName: \"kubernetes.io/projected/9d2dbde5-e102-48ee-a803-c14a844aaa9d-kube-api-access-4c545\") pod \"insights-runtime-extractor-56cpk\" (UID: \"9d2dbde5-e102-48ee-a803-c14a844aaa9d\") " pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.637623 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.637612 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z86ph\" (UniqueName: \"kubernetes.io/projected/ad85509c-9e79-4bdc-a069-235d4af67c62-kube-api-access-z86ph\") pod \"downloads-6dd789c8b7-mj8hx\" (UID: \"ad85509c-9e79-4bdc-a069-235d4af67c62\") " pod="openshift-console/downloads-6dd789c8b7-mj8hx" Nov 03 22:07:27.637802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.637788 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chqcd\" (UniqueName: \"kubernetes.io/projected/d9504538-9906-445e-960c-5c7d9e0e7598-kube-api-access-chqcd\") pod \"ingress-canary-g74dr\" (UID: \"d9504538-9906-445e-960c-5c7d9e0e7598\") " pod="openshift-ingress-canary/ingress-canary-g74dr" Nov 03 22:07:27.667901 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.667863 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:27.679967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.679893 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17697329-5597-4aff-a1fa-9c55b5a63e62-trusted-ca\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.679967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.679927 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb97h\" (UniqueName: \"kubernetes.io/projected/17697329-5597-4aff-a1fa-9c55b5a63e62-kube-api-access-xb97h\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.679967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.679945 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17697329-5597-4aff-a1fa-9c55b5a63e62-registry-certificates\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.679967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.679965 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5ed133-b829-4cf8-accf-9fbe62d82a31-config-volume\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.680255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.680237 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17697329-5597-4aff-a1fa-9c55b5a63e62-bound-sa-token\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.680304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.680274 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17697329-5597-4aff-a1fa-9c55b5a63e62-installation-pull-secrets\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.680349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.680314 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22lbt\" (UniqueName: \"kubernetes.io/projected/9f5ed133-b829-4cf8-accf-9fbe62d82a31-kube-api-access-22lbt\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.680349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.680345 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17697329-5597-4aff-a1fa-9c55b5a63e62-registry-tls\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.680421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.680363 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f5ed133-b829-4cf8-accf-9fbe62d82a31-metrics-tls\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.680421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.680382 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5ed133-b829-4cf8-accf-9fbe62d82a31-tmp-dir\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.680498 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.680477 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17697329-5597-4aff-a1fa-9c55b5a63e62-image-registry-private-configuration\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.680532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.680516 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17697329-5597-4aff-a1fa-9c55b5a63e62-ca-trust-extracted\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.681122 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.681099 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5ed133-b829-4cf8-accf-9fbe62d82a31-tmp-dir\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.681273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.681250 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17697329-5597-4aff-a1fa-9c55b5a63e62-ca-trust-extracted\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.681481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.681447 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5ed133-b829-4cf8-accf-9fbe62d82a31-config-volume\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.681596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.681550 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17697329-5597-4aff-a1fa-9c55b5a63e62-registry-certificates\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.681956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.681936 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17697329-5597-4aff-a1fa-9c55b5a63e62-trusted-ca\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.683580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.683552 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f5ed133-b829-4cf8-accf-9fbe62d82a31-metrics-tls\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.683735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.683715 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17697329-5597-4aff-a1fa-9c55b5a63e62-installation-pull-secrets\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.683808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.683742 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17697329-5597-4aff-a1fa-9c55b5a63e62-registry-tls\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.683868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.683832 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17697329-5597-4aff-a1fa-9c55b5a63e62-image-registry-private-configuration\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.688602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.688587 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mj8hx" Nov 03 22:07:27.701380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.701358 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g74dr" Nov 03 22:07:27.711536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.711507 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-56cpk" Nov 03 22:07:27.748279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.746273 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22lbt\" (UniqueName: \"kubernetes.io/projected/9f5ed133-b829-4cf8-accf-9fbe62d82a31-kube-api-access-22lbt\") pod \"dns-default-xgkft\" (UID: \"9f5ed133-b829-4cf8-accf-9fbe62d82a31\") " pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:27.748279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.746290 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17697329-5597-4aff-a1fa-9c55b5a63e62-bound-sa-token\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.748279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.746469 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb97h\" (UniqueName: \"kubernetes.io/projected/17697329-5597-4aff-a1fa-9c55b5a63e62-kube-api-access-xb97h\") pod \"image-registry-7d57db45fb-zszqm\" (UID: \"17697329-5597-4aff-a1fa-9c55b5a63e62\") " pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:27.789672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:27.789239 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:28.041932 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.041887 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:28.087062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.087030 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mj8hx"] Nov 03 22:07:28.089248 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:28.089213 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad85509c_9e79_4bdc_a069_235d4af67c62.slice/crio-30400eddade5fb70b9f2ad21774317c2ec62d8f85d11ef2c50cfd115dccd05c6 WatchSource:0}: Error finding container 30400eddade5fb70b9f2ad21774317c2ec62d8f85d11ef2c50cfd115dccd05c6: Status 404 returned error can't find the container with id 30400eddade5fb70b9f2ad21774317c2ec62d8f85d11ef2c50cfd115dccd05c6 Nov 03 22:07:28.090036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.089931 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:28.105030 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:28.105002 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9504538_9906_445e_960c_5c7d9e0e7598.slice/crio-dbfefc040268b844c30a5565cd80416179259e97218634bb968bb7f6ea701af0 WatchSource:0}: Error finding container dbfefc040268b844c30a5565cd80416179259e97218634bb968bb7f6ea701af0: Status 404 returned error can't find the container with id dbfefc040268b844c30a5565cd80416179259e97218634bb968bb7f6ea701af0 Nov 03 22:07:28.105382 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.105363 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g74dr"] Nov 03 22:07:28.107148 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:28.107131 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2dbde5_e102_48ee_a803_c14a844aaa9d.slice/crio-ea95574b4cb453b4632af23cd82d8a4fabedd15f1a6a1771324880392c2dc43f WatchSource:0}: Error finding container ea95574b4cb453b4632af23cd82d8a4fabedd15f1a6a1771324880392c2dc43f: Status 404 returned error can't find the container with id ea95574b4cb453b4632af23cd82d8a4fabedd15f1a6a1771324880392c2dc43f Nov 03 22:07:28.108095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.108072 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-56cpk"] Nov 03 22:07:28.165707 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:28.165675 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17697329_5597_4aff_a1fa_9c55b5a63e62.slice/crio-194d8f2580142440063e70b957b8a3f00feea2108bd56624a901f9f86629ef13 WatchSource:0}: Error finding container 194d8f2580142440063e70b957b8a3f00feea2108bd56624a901f9f86629ef13: Status 404 returned error can't find the container with id 194d8f2580142440063e70b957b8a3f00feea2108bd56624a901f9f86629ef13 Nov 03 22:07:28.167231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.167203 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d57db45fb-zszqm"] Nov 03 22:07:28.179922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.179891 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" event={"ID":"17697329-5597-4aff-a1fa-9c55b5a63e62","Type":"ContainerStarted","Data":"194d8f2580142440063e70b957b8a3f00feea2108bd56624a901f9f86629ef13"} Nov 03 22:07:28.181460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.181431 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56cpk" event={"ID":"9d2dbde5-e102-48ee-a803-c14a844aaa9d","Type":"ContainerStarted","Data":"255f981b60801933fd77b859ca3e3c8e841a6f953a6d108313693daa596ce858"} Nov 03 22:07:28.181568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.181492 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56cpk" event={"ID":"9d2dbde5-e102-48ee-a803-c14a844aaa9d","Type":"ContainerStarted","Data":"ea95574b4cb453b4632af23cd82d8a4fabedd15f1a6a1771324880392c2dc43f"} Nov 03 22:07:28.183130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.183083 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g74dr" event={"ID":"d9504538-9906-445e-960c-5c7d9e0e7598","Type":"ContainerStarted","Data":"dbfefc040268b844c30a5565cd80416179259e97218634bb968bb7f6ea701af0"} Nov 03 22:07:28.185676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.185622 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mj8hx" event={"ID":"ad85509c-9e79-4bdc-a069-235d4af67c62","Type":"ContainerStarted","Data":"30400eddade5fb70b9f2ad21774317c2ec62d8f85d11ef2c50cfd115dccd05c6"} Nov 03 22:07:28.266610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.266572 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:28.405905 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:28.405869 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5ed133_b829_4cf8_accf_9fbe62d82a31.slice/crio-c8c6c0466804ea862ce1ad362c08791d8d3dc08b26e8da40e5c177a24af3fd17 WatchSource:0}: Error finding container c8c6c0466804ea862ce1ad362c08791d8d3dc08b26e8da40e5c177a24af3fd17: Status 404 returned error can't find the container with id c8c6c0466804ea862ce1ad362c08791d8d3dc08b26e8da40e5c177a24af3fd17 Nov 03 22:07:28.408681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.406472 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xgkft"] Nov 03 22:07:28.591075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:28.591018 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:29.000506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:29.000465 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:29.169943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:29.169907 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:29.189367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:29.189330 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" event={"ID":"17697329-5597-4aff-a1fa-9c55b5a63e62","Type":"ContainerStarted","Data":"bc01b17fce88716544c08f539e21c6c151b044278dfc3e95570aa2a92624009b"} Nov 03 22:07:29.190702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:29.190674 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xgkft" event={"ID":"9f5ed133-b829-4cf8-accf-9fbe62d82a31","Type":"ContainerStarted","Data":"c8c6c0466804ea862ce1ad362c08791d8d3dc08b26e8da40e5c177a24af3fd17"} Nov 03 22:07:29.308036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:29.307900 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" podStartSLOduration=2.307881022 podStartE2EDuration="2.307881022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:29.307054046 +0000 UTC m=+6.287787700" watchObservedRunningTime="2026-11-03 22:07:29.307881022 +0000 UTC m=+6.288614677" Nov 03 22:07:30.196787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:30.196758 4298 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:07:30.197242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:30.196984 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56cpk" event={"ID":"9d2dbde5-e102-48ee-a803-c14a844aaa9d","Type":"ContainerStarted","Data":"88a22570d8da507743996902c2f6abb11d4d5ddfe0d88c045701e80b09953c84"} Nov 03 22:07:30.691999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:30.691967 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6q9pz_d381f494-4ddc-4d70-9d0f-f27d6b3a7d8d/dns-node-resolver/0.log" Nov 03 22:07:31.009269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:31.009243 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:31.009501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:31.009487 4298 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:07:31.692343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:31.692258 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d57db45fb-zszqm_17697329-5597-4aff-a1fa-9c55b5a63e62/registry/0.log" Nov 03 22:07:31.891849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:31.891755 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k98wl_76b201e3-7c0b-4810-9baa-f6e054cf0826/node-ca/0.log" Nov 03 22:07:32.059812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.059782 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:32.069168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.069139 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bc6474b6-98qng"] Nov 03 22:07:32.072243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.072220 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.126504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.126459 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 22:07:32.126620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.126573 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 22:07:32.127397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.127369 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc6474b6-98qng"] Nov 03 22:07:32.177815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.177788 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 22:07:32.177934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.177916 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 22:07:32.178014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.177999 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lr7tn\"" Nov 03 22:07:32.178390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.178367 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 22:07:32.207874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.207789 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56cpk" event={"ID":"9d2dbde5-e102-48ee-a803-c14a844aaa9d","Type":"ContainerStarted","Data":"4d86eb704cd6d26dcb7eb6eadfc329b24c53e67f21eb86c14e0cf2ed64b0b526"} Nov 03 22:07:32.209899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.209869 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g74dr" event={"ID":"d9504538-9906-445e-960c-5c7d9e0e7598","Type":"ContainerStarted","Data":"12e6cd6bced4d66c8d6d71eab213194ddb5b524e156a6b276f7c0c17e5b02af4"} Nov 03 22:07:32.212765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.212737 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xgkft" event={"ID":"9f5ed133-b829-4cf8-accf-9fbe62d82a31","Type":"ContainerStarted","Data":"87bd8fca581f4e2abce5d4654fbc969e171bf5c398034e209589d74838512b6a"} Nov 03 22:07:32.212874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.212790 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xgkft" event={"ID":"9f5ed133-b829-4cf8-accf-9fbe62d82a31","Type":"ContainerStarted","Data":"e77fe0e38b85de038b9a9dfee97fab8c968977a6dc679b1b7ba9742cbd0e1579"} Nov 03 22:07:32.213296 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.213276 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:32.217751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.217730 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-service-ca\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.217830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.217760 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09683956-f309-4c54-b6cb-1ff43888ccd0-tmp\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.217830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.217778 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-serving-cert\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.217830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.217794 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-console-config\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.217830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.217817 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-oauth-config\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.217969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.217863 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-oauth-serving-cert\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.217969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.217879 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqwfj\" (UniqueName: \"kubernetes.io/projected/09683956-f309-4c54-b6cb-1ff43888ccd0-kube-api-access-bqwfj\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.266417 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.266380 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:32.319165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.318416 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-oauth-serving-cert\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.319165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.318461 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqwfj\" (UniqueName: \"kubernetes.io/projected/09683956-f309-4c54-b6cb-1ff43888ccd0-kube-api-access-bqwfj\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.319165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.318509 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-service-ca\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.319165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.318532 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09683956-f309-4c54-b6cb-1ff43888ccd0-tmp\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.319165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.318558 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-serving-cert\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.319165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.318579 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-console-config\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.319165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.318639 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-oauth-config\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.319674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.319467 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-service-ca\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.319674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.319533 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-oauth-serving-cert\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.319674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.319563 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-console-config\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.319951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.319923 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09683956-f309-4c54-b6cb-1ff43888ccd0-tmp\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.321514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.321489 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-oauth-config\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.322357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.322333 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-serving-cert\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.362627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.362562 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-56cpk" podStartSLOduration=1.5195475489999999 podStartE2EDuration="5.36254697s" totalImagesPullingTime="3.842999421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:07:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:32.362321786 +0000 UTC m=+9.343055438" watchObservedRunningTime="2026-11-03 22:07:32.36254697 +0000 UTC m=+9.343280620" Nov 03 22:07:32.377574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.377545 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqwfj\" (UniqueName: \"kubernetes.io/projected/09683956-f309-4c54-b6cb-1ff43888ccd0-kube-api-access-bqwfj\") pod \"console-5bc6474b6-98qng\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.385219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.385187 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:32.480138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.479787 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xgkft" podStartSLOduration=2.765540004 podStartE2EDuration="5.479769212s" totalImagesPullingTime="2.714229208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:32.479365443 +0000 UTC m=+9.460099096" watchObservedRunningTime="2026-11-03 22:07:32.479769212 +0000 UTC m=+9.460502864" Nov 03 22:07:32.596348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.596245 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g74dr" podStartSLOduration=1.602878357 podStartE2EDuration="5.596227911s" totalImagesPullingTime="3.993349554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:32.595984142 +0000 UTC m=+9.576717795" watchObservedRunningTime="2026-11-03 22:07:32.596227911 +0000 UTC m=+9.576961563" Nov 03 22:07:32.691542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.691513 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g74dr_d9504538-9906-445e-960c-5c7d9e0e7598/serve-healthcheck-canary/0.log" Nov 03 22:07:32.804383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:32.802812 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc6474b6-98qng"] Nov 03 22:07:32.804855 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:32.804763 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09683956_f309_4c54_b6cb_1ff43888ccd0.slice/crio-75600bd178c8ec359987e35ef5ba31155180cfa985299a43362595e88be98ca4 WatchSource:0}: Error finding container 75600bd178c8ec359987e35ef5ba31155180cfa985299a43362595e88be98ca4: Status 404 returned error can't find the container with id 75600bd178c8ec359987e35ef5ba31155180cfa985299a43362595e88be98ca4 Nov 03 22:07:33.220154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:33.220114 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc6474b6-98qng" event={"ID":"09683956-f309-4c54-b6cb-1ff43888ccd0","Type":"ContainerStarted","Data":"75600bd178c8ec359987e35ef5ba31155180cfa985299a43362595e88be98ca4"} Nov 03 22:07:34.020539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:34.020504 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:34.070260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:34.070225 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:34.194946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:34.194912 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:34.365058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:34.364967 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:34.825854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:34.825819 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:35.010481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:35.010426 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:35.192033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:35.191951 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:35.364945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:35.364913 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:37.236779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:37.236736 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc6474b6-98qng" event={"ID":"09683956-f309-4c54-b6cb-1ff43888ccd0","Type":"ContainerStarted","Data":"787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff"} Nov 03 22:07:37.354459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:37.354395 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5bc6474b6-98qng" podStartSLOduration=1.72983312 podStartE2EDuration="5.35438102s" totalImagesPullingTime="3.6245479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:37.353586977 +0000 UTC m=+14.334320631" watchObservedRunningTime="2026-11-03 22:07:37.35438102 +0000 UTC m=+14.335114672" Nov 03 22:07:37.621252 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:37.621177 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:37.801221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:37.801189 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:38.375587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:38.375551 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:38.548617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:38.548581 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:38.551519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:38.551498 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:38.557301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:38.557273 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:38.721140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:38.721113 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:38.895699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:38.895668 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:40.251291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:40.251259 4298 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:07:41.875734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:41.875697 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:41.931849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:41.931811 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:41.932017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:41.931937 4298 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:07:41.936439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:41.936413 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:07:42.051310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:42.051274 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:42.299305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:42.299275 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5gw2b" Nov 03 22:07:43.229248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:43.229219 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xgkft" Nov 03 22:07:43.412080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:43.412046 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:43.633849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:43.633772 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:44.076289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:44.076258 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d57db45fb-zszqm" Nov 03 22:07:44.415774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:44.415687 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:44.596233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:44.596200 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:44.784216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:44.784181 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:44.986394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:44.986355 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:45.177081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.176994 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:45.358303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.358271 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:45.452278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.452249 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m54ln"] Nov 03 22:07:45.470100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.470073 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.523241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.523221 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:07:45.523487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.523468 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:07:45.523487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.523481 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:07:45.523609 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.523475 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:07:45.535800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.535771 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:45.574611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.574583 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:07:45.574777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.574630 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xzmgl\"" Nov 03 22:07:45.574777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.574687 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:07:45.623546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.623513 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-accelerators-collector-config\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.623546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.623549 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cd23ca2-0297-4e50-b026-3208a1244520-root\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.623800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.623571 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cd23ca2-0297-4e50-b026-3208a1244520-metrics-client-ca\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.623800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.623743 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-wtmp\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.623800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.623785 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd23ca2-0297-4e50-b026-3208a1244520-sys\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.623933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.623831 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-textfile\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.623933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.623886 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-tls\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.624008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.623938 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.624008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.623967 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7cg5\" (UniqueName: \"kubernetes.io/projected/8cd23ca2-0297-4e50-b026-3208a1244520-kube-api-access-x7cg5\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.724851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.724764 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-accelerators-collector-config\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.724851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.724806 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cd23ca2-0297-4e50-b026-3208a1244520-root\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.724851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.724828 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cd23ca2-0297-4e50-b026-3208a1244520-metrics-client-ca\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.725117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.724866 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-wtmp\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.725117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.724885 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd23ca2-0297-4e50-b026-3208a1244520-sys\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.725117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.724909 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-textfile\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.725117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.724917 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cd23ca2-0297-4e50-b026-3208a1244520-root\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.725117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.724936 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-tls\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.725117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.724966 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.725117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.725002 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7cg5\" (UniqueName: \"kubernetes.io/projected/8cd23ca2-0297-4e50-b026-3208a1244520-kube-api-access-x7cg5\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.725117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.725052 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-wtmp\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.725490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.724969 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd23ca2-0297-4e50-b026-3208a1244520-sys\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.725768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.725712 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-textfile\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.725944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.725921 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-accelerators-collector-config\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.726070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.726050 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cd23ca2-0297-4e50-b026-3208a1244520-metrics-client-ca\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.727519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.727494 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-tls\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.727747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.727722 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cd23ca2-0297-4e50-b026-3208a1244520-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.783192 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.783160 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7cg5\" (UniqueName: \"kubernetes.io/projected/8cd23ca2-0297-4e50-b026-3208a1244520-kube-api-access-x7cg5\") pod \"node-exporter-m54ln\" (UID: \"8cd23ca2-0297-4e50-b026-3208a1244520\") " pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.809144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.809105 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m54ln" Nov 03 22:07:45.816863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:45.816829 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:45.838763 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:45.838733 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd23ca2_0297_4e50_b026_3208a1244520.slice/crio-aa93cc222064fc8930050a7dcca1e18f578851ff8304dd346e4f5a836ab7d464 WatchSource:0}: Error finding container aa93cc222064fc8930050a7dcca1e18f578851ff8304dd346e4f5a836ab7d464: Status 404 returned error can't find the container with id aa93cc222064fc8930050a7dcca1e18f578851ff8304dd346e4f5a836ab7d464 Nov 03 22:07:46.000486 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:46.000381 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:46.273807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:46.273681 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mj8hx" event={"ID":"ad85509c-9e79-4bdc-a069-235d4af67c62","Type":"ContainerStarted","Data":"4f7c3ab201b80257b06380bba147616bf2635320655ea10b8fc8da678f762fd1"} Nov 03 22:07:46.273807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:46.273806 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-mj8hx" Nov 03 22:07:46.275950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:46.275256 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m54ln" event={"ID":"8cd23ca2-0297-4e50-b026-3208a1244520","Type":"ContainerStarted","Data":"aa93cc222064fc8930050a7dcca1e18f578851ff8304dd346e4f5a836ab7d464"} Nov 03 22:07:46.291289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:46.291257 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-mj8hx" Nov 03 22:07:46.390335 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:46.390279 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-mj8hx" podStartSLOduration=1.96254245 podStartE2EDuration="19.390263382s" totalImagesPullingTime="17.427720932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:46.389227015 +0000 UTC m=+23.369960667" watchObservedRunningTime="2026-11-03 22:07:46.390263382 +0000 UTC m=+23.370997068" Nov 03 22:07:46.727394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:46.727356 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:46.902477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:46.902433 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:47.083052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.082973 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:47.563700 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:07:47.563565 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:07:47.735928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.735893 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:47.782247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.782206 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:07:47.811018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.810984 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.833858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.833824 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:07:47.883548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.883388 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 22:07:47.883548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.883392 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c8rtb\"" Nov 03 22:07:47.883548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.883387 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 22:07:47.883548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.883456 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 22:07:47.883548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.883458 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 22:07:47.883548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.883486 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 22:07:47.883548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.883461 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 22:07:47.883548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.883521 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 22:07:47.883548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.883525 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 22:07:47.941459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.941425 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.941459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.941463 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.941742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.941496 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-web-config\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.941742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.941660 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/058b48e3-5571-4d76-a4a0-efb0eced09ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.941742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.941709 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/058b48e3-5571-4d76-a4a0-efb0eced09ee-config-out\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.941900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.941744 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.941900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.941820 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/058b48e3-5571-4d76-a4a0-efb0eced09ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.941900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.941865 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.942018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.941907 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42lrx\" (UniqueName: \"kubernetes.io/projected/058b48e3-5571-4d76-a4a0-efb0eced09ee-kube-api-access-42lrx\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.942018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.941946 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.942018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.941969 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.942144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.942040 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/058b48e3-5571-4d76-a4a0-efb0eced09ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.942144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.942063 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/058b48e3-5571-4d76-a4a0-efb0eced09ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:47.968597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:47.968558 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 22:07:48.043187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043137 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-web-config\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043190 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/058b48e3-5571-4d76-a4a0-efb0eced09ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043229 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/058b48e3-5571-4d76-a4a0-efb0eced09ee-config-out\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043252 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043275 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/058b48e3-5571-4d76-a4a0-efb0eced09ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043298 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043320 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42lrx\" (UniqueName: \"kubernetes.io/projected/058b48e3-5571-4d76-a4a0-efb0eced09ee-kube-api-access-42lrx\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043348 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043383 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043430 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/058b48e3-5571-4d76-a4a0-efb0eced09ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043455 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/058b48e3-5571-4d76-a4a0-efb0eced09ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043488 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.043801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.043516 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.045598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.044357 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/058b48e3-5571-4d76-a4a0-efb0eced09ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.045598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.044468 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/058b48e3-5571-4d76-a4a0-efb0eced09ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.046762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.046517 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/058b48e3-5571-4d76-a4a0-efb0eced09ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.048735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.048196 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.048735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.048330 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.048735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.048681 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.048735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.048688 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/058b48e3-5571-4d76-a4a0-efb0eced09ee-config-out\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.048987 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.048964 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-web-config\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.049067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.049044 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.050458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.050435 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.050458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.050455 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/058b48e3-5571-4d76-a4a0-efb0eced09ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.051138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.051118 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/058b48e3-5571-4d76-a4a0-efb0eced09ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.104779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.104749 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42lrx\" (UniqueName: \"kubernetes.io/projected/058b48e3-5571-4d76-a4a0-efb0eced09ee-kube-api-access-42lrx\") pod \"alertmanager-main-0\" (UID: \"058b48e3-5571-4d76-a4a0-efb0eced09ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.274385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.274333 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:07:48.290164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.290107 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m54ln" event={"ID":"8cd23ca2-0297-4e50-b026-3208a1244520","Type":"ContainerStarted","Data":"972d8ffea9b23c11de25d435c76dd10c8c271a5038cc2bd5ddc246d1ac4f5b51"} Nov 03 22:07:48.806233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.806206 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:07:48.894559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:48.894512 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:49.067802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:49.067710 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:49.298280 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:49.298141 4298 generic.go:356] "Generic (PLEG): container finished" podID="8cd23ca2-0297-4e50-b026-3208a1244520" containerID="972d8ffea9b23c11de25d435c76dd10c8c271a5038cc2bd5ddc246d1ac4f5b51" exitCode=0 Nov 03 22:07:49.298474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:49.298341 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m54ln" event={"ID":"8cd23ca2-0297-4e50-b026-3208a1244520","Type":"ContainerDied","Data":"972d8ffea9b23c11de25d435c76dd10c8c271a5038cc2bd5ddc246d1ac4f5b51"} Nov 03 22:07:49.298474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:49.298408 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m54ln" event={"ID":"8cd23ca2-0297-4e50-b026-3208a1244520","Type":"ContainerStarted","Data":"9f0aecee8024a86082b83dc760412ca621d732782762d9c5ada35bfcaa25d3c3"} Nov 03 22:07:49.298474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:49.298436 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m54ln" event={"ID":"8cd23ca2-0297-4e50-b026-3208a1244520","Type":"ContainerStarted","Data":"02d17e5ece3c3955ea59e3e043049bed439922dce96e9045d6736abbdd09854e"} Nov 03 22:07:49.301666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:49.301280 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"058b48e3-5571-4d76-a4a0-efb0eced09ee","Type":"ContainerStarted","Data":"6b360c7a95d37963954a71c7c74bd7a8e26eeeef594626bffc40fa7fdc707138"} Nov 03 22:07:49.418578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:49.416687 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m54ln" podStartSLOduration=2.098344587 podStartE2EDuration="4.41666771s" totalImagesPullingTime="2.241496234s" totalInitContainerRuntime="76.826889ms" isStatefulPod=true podCreationTimestamp="2026-11-03 22:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:49.416349339 +0000 UTC m=+26.397082991" watchObservedRunningTime="2026-11-03 22:07:49.41666771 +0000 UTC m=+26.397401365" Nov 03 22:07:49.508284 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:49.508249 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:49.682472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:49.682391 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:49.867536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:49.867502 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:50.041585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.041514 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:50.106923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.105727 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7"] Nov 03 22:07:50.128800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.128696 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" Nov 03 22:07:50.168733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.168277 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7"] Nov 03 22:07:50.205362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.205308 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cj9c2\"" Nov 03 22:07:50.205362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.205360 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 22:07:50.213415 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.213320 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:50.270450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.270411 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sns8w\" (UniqueName: \"kubernetes.io/projected/f6231d72-95c6-44f1-adde-246081a3665c-kube-api-access-sns8w\") pod \"monitoring-plugin-77f4f76594-lwmf7\" (UID: \"f6231d72-95c6-44f1-adde-246081a3665c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" Nov 03 22:07:50.270618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.270511 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6231d72-95c6-44f1-adde-246081a3665c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-lwmf7\" (UID: \"f6231d72-95c6-44f1-adde-246081a3665c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" Nov 03 22:07:50.371955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.371703 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sns8w\" (UniqueName: \"kubernetes.io/projected/f6231d72-95c6-44f1-adde-246081a3665c-kube-api-access-sns8w\") pod \"monitoring-plugin-77f4f76594-lwmf7\" (UID: \"f6231d72-95c6-44f1-adde-246081a3665c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" Nov 03 22:07:50.371955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.371756 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6231d72-95c6-44f1-adde-246081a3665c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-lwmf7\" (UID: \"f6231d72-95c6-44f1-adde-246081a3665c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" Nov 03 22:07:50.396378 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.375668 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6231d72-95c6-44f1-adde-246081a3665c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-lwmf7\" (UID: \"f6231d72-95c6-44f1-adde-246081a3665c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" Nov 03 22:07:50.396378 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.388122 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:50.432350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.432311 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sns8w\" (UniqueName: \"kubernetes.io/projected/f6231d72-95c6-44f1-adde-246081a3665c-kube-api-access-sns8w\") pod \"monitoring-plugin-77f4f76594-lwmf7\" (UID: \"f6231d72-95c6-44f1-adde-246081a3665c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" Nov 03 22:07:50.462136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.462092 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" Nov 03 22:07:50.862620 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:50.862587 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6231d72_95c6_44f1_adde_246081a3665c.slice/crio-7a2c162038044f5236b774cfd494233d3f07a36d10c8508018aa10f59bcc3438 WatchSource:0}: Error finding container 7a2c162038044f5236b774cfd494233d3f07a36d10c8508018aa10f59bcc3438: Status 404 returned error can't find the container with id 7a2c162038044f5236b774cfd494233d3f07a36d10c8508018aa10f59bcc3438 Nov 03 22:07:50.863294 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:50.863265 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7"] Nov 03 22:07:51.319574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:51.319061 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"058b48e3-5571-4d76-a4a0-efb0eced09ee","Type":"ContainerStarted","Data":"6a3b669990baa5f6dc676b411a3415e2a067ba565ed08187e14e64181459a8f8"} Nov 03 22:07:51.320546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:51.320513 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" event={"ID":"f6231d72-95c6-44f1-adde-246081a3665c","Type":"ContainerStarted","Data":"7a2c162038044f5236b774cfd494233d3f07a36d10c8508018aa10f59bcc3438"} Nov 03 22:07:52.327959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:52.327787 4298 generic.go:356] "Generic (PLEG): container finished" podID="058b48e3-5571-4d76-a4a0-efb0eced09ee" containerID="6a3b669990baa5f6dc676b411a3415e2a067ba565ed08187e14e64181459a8f8" exitCode=0 Nov 03 22:07:52.327959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:52.327868 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"058b48e3-5571-4d76-a4a0-efb0eced09ee","Type":"ContainerDied","Data":"6a3b669990baa5f6dc676b411a3415e2a067ba565ed08187e14e64181459a8f8"} Nov 03 22:07:52.513056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:52.513023 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:52.747004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:52.746970 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:54.337749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.337700 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" event={"ID":"f6231d72-95c6-44f1-adde-246081a3665c","Type":"ContainerStarted","Data":"8118d861830ab8fe06482cf18e7b126612c256105193598689e19b795ebc1b4c"} Nov 03 22:07:54.338271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.337883 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" Nov 03 22:07:54.343313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.343278 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" Nov 03 22:07:54.455363 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.455300 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwmf7" podStartSLOduration=1.875523515 podStartE2EDuration="4.455281754s" totalImagesPullingTime="2.579758239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:54.45440911 +0000 UTC m=+31.435142766" watchObservedRunningTime="2026-11-03 22:07:54.455281754 +0000 UTC m=+31.436015407" Nov 03 22:07:54.484390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.484352 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:07:54.493627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.493597 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:54.498546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.498518 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.549199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.549161 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:07:54.553181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.553111 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 22:07:54.553181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.553132 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 22:07:54.553181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.553116 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 22:07:54.553181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.553157 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 22:07:54.553181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.553111 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 22:07:54.553463 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.553190 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 22:07:54.553463 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.553264 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 22:07:54.553463 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.553309 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jlpf5\"" Nov 03 22:07:54.605009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.604934 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-pcob6ei5ut7v\"" Nov 03 22:07:54.605009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.604987 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 22:07:54.605315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.605005 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 22:07:54.605315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.604992 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 22:07:54.605315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.605054 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 22:07:54.605315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.605097 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 22:07:54.605315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.605196 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 22:07:54.607419 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607396 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.607537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607426 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.607537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607444 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.607537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607500 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.607732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607623 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.607732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607667 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.607732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607690 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.607732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607713 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.607902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607736 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.607902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607809 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.607902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607874 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.607902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607894 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9rvn\" (UniqueName: \"kubernetes.io/projected/70e4fb16-2a0d-48f5-b326-f7ce057042f0-kube-api-access-b9rvn\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.608093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607911 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.608093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607935 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.608093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.607982 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.608093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.608019 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-config\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.608093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.608043 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70e4fb16-2a0d-48f5-b326-f7ce057042f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.608093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.608069 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.608555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.608145 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.608555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.608283 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70e4fb16-2a0d-48f5-b326-f7ce057042f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.655343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.655310 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 22:07:54.655549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.655485 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 22:07:54.709074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709034 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709083 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9rvn\" (UniqueName: \"kubernetes.io/projected/70e4fb16-2a0d-48f5-b326-f7ce057042f0-kube-api-access-b9rvn\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709111 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709260 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709296 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709329 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-config\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709552 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70e4fb16-2a0d-48f5-b326-f7ce057042f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709590 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709669 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709714 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70e4fb16-2a0d-48f5-b326-f7ce057042f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709760 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709786 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709811 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709820 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709848 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709879 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.709930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709910 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.710173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709937 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.710173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709965 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.710173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.709991 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.710173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.710035 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.710173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.710051 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.710498 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.710286 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.710708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.710687 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.713036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.713012 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-config\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.713261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.713232 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70e4fb16-2a0d-48f5-b326-f7ce057042f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.713369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.713310 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.713762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.713738 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70e4fb16-2a0d-48f5-b326-f7ce057042f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.714904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.714025 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.714904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.714535 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.715109 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.715076 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.715550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.715524 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.715550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.715537 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.715930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.715911 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.716313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.716283 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.716619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.716594 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70e4fb16-2a0d-48f5-b326-f7ce057042f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.716920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.716903 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.717325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.717307 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.717325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.717314 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70e4fb16-2a0d-48f5-b326-f7ce057042f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.767480 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.767449 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9rvn\" (UniqueName: \"kubernetes.io/projected/70e4fb16-2a0d-48f5-b326-f7ce057042f0-kube-api-access-b9rvn\") pod \"prometheus-k8s-0\" (UID: \"70e4fb16-2a0d-48f5-b326-f7ce057042f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:54.906998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:54.906922 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:07:55.170846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:55.170761 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bljnw" Nov 03 22:07:55.370167 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:07:55.370123 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e4fb16_2a0d_48f5_b326_f7ce057042f0.slice/crio-fc8f65b4bab211d19655316030dadcab60f4682728621c87d15482788388f659 WatchSource:0}: Error finding container fc8f65b4bab211d19655316030dadcab60f4682728621c87d15482788388f659: Status 404 returned error can't find the container with id fc8f65b4bab211d19655316030dadcab60f4682728621c87d15482788388f659 Nov 03 22:07:55.371809 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:55.371782 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:07:56.347117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:56.347083 4298 generic.go:356] "Generic (PLEG): container finished" podID="70e4fb16-2a0d-48f5-b326-f7ce057042f0" containerID="add457a045e359c789be549ebbf01300fa06c824c1fc1854e7292c0ad9eb9bae" exitCode=0 Nov 03 22:07:56.347355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:56.347181 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70e4fb16-2a0d-48f5-b326-f7ce057042f0","Type":"ContainerDied","Data":"add457a045e359c789be549ebbf01300fa06c824c1fc1854e7292c0ad9eb9bae"} Nov 03 22:07:56.347355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:56.347276 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70e4fb16-2a0d-48f5-b326-f7ce057042f0","Type":"ContainerStarted","Data":"fc8f65b4bab211d19655316030dadcab60f4682728621c87d15482788388f659"} Nov 03 22:07:56.350140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:56.350115 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"058b48e3-5571-4d76-a4a0-efb0eced09ee","Type":"ContainerStarted","Data":"2d28c996a62e705ec9eecfc8dcfd25648ec0ea15bf0f6d80d2b8ae4cff050adc"} Nov 03 22:07:56.350234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:56.350187 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"058b48e3-5571-4d76-a4a0-efb0eced09ee","Type":"ContainerStarted","Data":"31c9e721a0fb3531c2e215ab1f122fe776b2822778050251a29919cdd67625b6"} Nov 03 22:07:56.350234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:56.350226 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"058b48e3-5571-4d76-a4a0-efb0eced09ee","Type":"ContainerStarted","Data":"0613facdf80d3a4d9e60a62eaf252bbf3ab39b8f4843e674671d726e4143a557"} Nov 03 22:07:56.890259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:56.890225 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:57.095534 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:57.095501 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:57.356771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:57.356734 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"058b48e3-5571-4d76-a4a0-efb0eced09ee","Type":"ContainerStarted","Data":"11327d0a6b22cb510b3e26c8eae7ffffcd3f6f59c7f1ba11e4db256dcc7518a4"} Nov 03 22:07:57.356940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:57.356801 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"058b48e3-5571-4d76-a4a0-efb0eced09ee","Type":"ContainerStarted","Data":"e1c78b67b464aae07449387e69c5e03b6cd8427f32d166d806dd7fe5f4c9063d"} Nov 03 22:07:59.368267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:59.368230 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"058b48e3-5571-4d76-a4a0-efb0eced09ee","Type":"ContainerStarted","Data":"ecc9921493f83e338b48e2c416f11048d7f3469acb646af4bdb6b7f27d79eff9"} Nov 03 22:07:59.646360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:07:59.646258 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.747543415 podStartE2EDuration="12.646239746s" totalImagesPullingTime="7.877591425s" totalInitContainerRuntime="21.104906ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:59.645224409 +0000 UTC m=+36.625958064" watchObservedRunningTime="2026-11-03 22:07:59.646239746 +0000 UTC m=+36.626973398" Nov 03 22:08:00.393924 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.393879 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-74d55f589f-dwsmd"] Nov 03 22:08:00.408654 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.408626 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.423496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.423473 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:00.455696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.455663 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d55f589f-dwsmd"] Nov 03 22:08:00.456128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.456106 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-config\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.456204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.456146 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-service-ca\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.456263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.456217 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a05ce10-05cb-433d-aa03-b6e9a4190eac-tmp\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.456712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.456493 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-serving-cert\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.456712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.456695 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-trusted-ca-bundle\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.456885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.456760 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz5fk\" (UniqueName: \"kubernetes.io/projected/0a05ce10-05cb-433d-aa03-b6e9a4190eac-kube-api-access-dz5fk\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.456885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.456825 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-oauth-config\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.456885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.456850 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-oauth-serving-cert\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.557913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.557882 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-config\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.557913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.557915 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-service-ca\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.558156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.557946 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a05ce10-05cb-433d-aa03-b6e9a4190eac-tmp\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.558156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.557976 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-serving-cert\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.558156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.557999 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-trusted-ca-bundle\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.558156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.558017 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz5fk\" (UniqueName: \"kubernetes.io/projected/0a05ce10-05cb-433d-aa03-b6e9a4190eac-kube-api-access-dz5fk\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.558156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.558034 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-oauth-config\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.558156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.558050 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-oauth-serving-cert\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.558464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.558427 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a05ce10-05cb-433d-aa03-b6e9a4190eac-tmp\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.558721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.558699 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-service-ca\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.558825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.558737 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-oauth-serving-cert\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.558825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.558749 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-config\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.560478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.560459 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-serving-cert\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.560561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.560525 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-oauth-config\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.607475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.607435 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 22:08:00.609108 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.609087 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-trusted-ca-bundle\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.619163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.619097 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz5fk\" (UniqueName: \"kubernetes.io/projected/0a05ce10-05cb-433d-aa03-b6e9a4190eac-kube-api-access-dz5fk\") pod \"console-74d55f589f-dwsmd\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:00.620340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.620317 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:00.740157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:00.740129 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:01.182114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:01.182077 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d55f589f-dwsmd"] Nov 03 22:08:01.182842 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:08:01.182814 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a05ce10_05cb_433d_aa03_b6e9a4190eac.slice/crio-209bf9cf83d911aa33a534460f2b68d565ed6704e8733f74bfd3c46cf675d9c3 WatchSource:0}: Error finding container 209bf9cf83d911aa33a534460f2b68d565ed6704e8733f74bfd3c46cf675d9c3: Status 404 returned error can't find the container with id 209bf9cf83d911aa33a534460f2b68d565ed6704e8733f74bfd3c46cf675d9c3 Nov 03 22:08:01.283863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:01.283832 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:01.378506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:01.378468 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70e4fb16-2a0d-48f5-b326-f7ce057042f0","Type":"ContainerStarted","Data":"945e0b4224dada7b03cb4516e40a427871414069f0ac3a5d014d8a9e6e7fdd39"} Nov 03 22:08:01.378831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:01.378798 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70e4fb16-2a0d-48f5-b326-f7ce057042f0","Type":"ContainerStarted","Data":"d53f15ff374710d796f3ef099b4596471a99ba935bf68569a409d96889b9150a"} Nov 03 22:08:01.380831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:01.380801 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d55f589f-dwsmd" event={"ID":"0a05ce10-05cb-433d-aa03-b6e9a4190eac","Type":"ContainerStarted","Data":"eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0"} Nov 03 22:08:01.380944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:01.380855 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d55f589f-dwsmd" event={"ID":"0a05ce10-05cb-433d-aa03-b6e9a4190eac","Type":"ContainerStarted","Data":"209bf9cf83d911aa33a534460f2b68d565ed6704e8733f74bfd3c46cf675d9c3"} Nov 03 22:08:01.462538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:01.462508 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:01.497707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:01.497619 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-74d55f589f-dwsmd" podStartSLOduration=1.497598607 podStartE2EDuration="1.497598607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:08:01.496818328 +0000 UTC m=+38.477551978" watchObservedRunningTime="2026-11-03 22:08:01.497598607 +0000 UTC m=+38.478332260" Nov 03 22:08:01.816991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:01.816935 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:02.001292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:02.001254 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:02.514944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:02.514901 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:02.731046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:02.731008 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:03.945624 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:03.945591 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:04.199224 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:04.199173 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:04.397989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:04.397956 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70e4fb16-2a0d-48f5-b326-f7ce057042f0","Type":"ContainerStarted","Data":"03f14952640e1bfcf684efb572bcb50dffdc372f4aaff2f1068c5af469e6a04a"} Nov 03 22:08:04.398146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:04.398046 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70e4fb16-2a0d-48f5-b326-f7ce057042f0","Type":"ContainerStarted","Data":"4fc2e946df0e92f283923871a3764e7c369d901fbcc8c83f441a2230e30c5e3c"} Nov 03 22:08:04.398146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:04.398077 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70e4fb16-2a0d-48f5-b326-f7ce057042f0","Type":"ContainerStarted","Data":"3186b474450b6084537740a5cde539ba0ae16edda73874e0d4b851568212f334"} Nov 03 22:08:04.398146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:04.398101 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70e4fb16-2a0d-48f5-b326-f7ce057042f0","Type":"ContainerStarted","Data":"3e8f8756f7baa1aedd0069045f11a3e14fd39a842022a59d7a898a05649f25f7"} Nov 03 22:08:04.578505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:04.578401 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.155238119 podStartE2EDuration="10.578383972s" totalImagesPullingTime="7.400933562s" totalInitContainerRuntime="22.212291ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:08:04.577014671 +0000 UTC m=+41.557748324" watchObservedRunningTime="2026-11-03 22:08:04.578383972 +0000 UTC m=+41.559117626" Nov 03 22:08:04.907844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:04.907756 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:08:04.929703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:04.929668 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:05.130961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:05.130926 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:10.740530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:10.740491 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:10.741045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:10.740553 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:10.745399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:10.745376 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:11.425117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:11.425091 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:08:11.595328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:11.595293 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc6474b6-98qng"] Nov 03 22:08:11.885866 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:11.885830 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:12.065202 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:12.065167 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:13.274979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:13.274935 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:17.550930 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:08:17.550894 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:08:25.706011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:25.705978 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:25.881343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:25.881309 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:26.887088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:26.887055 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:27.074568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:27.074530 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:34.877374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:34.877330 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:08:34.929624 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:34.929599 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:08:35.517602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:35.517575 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:08:36.614939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.614874 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5bc6474b6-98qng" podUID="09683956-f309-4c54-b6cb-1ff43888ccd0" containerName="console" containerID="cri-o://787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff" gracePeriod=15 Nov 03 22:08:36.858711 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.858680 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:08:36.971478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.971446 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-console-config\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-console-config\") pod \"09683956-f309-4c54-b6cb-1ff43888ccd0\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " Nov 03 22:08:36.971693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.971502 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09683956-f309-4c54-b6cb-1ff43888ccd0-tmp\" (UniqueName: \"kubernetes.io/empty-dir/09683956-f309-4c54-b6cb-1ff43888ccd0-tmp\") pod \"09683956-f309-4c54-b6cb-1ff43888ccd0\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " Nov 03 22:08:36.971693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.971550 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-serving-cert\") pod \"09683956-f309-4c54-b6cb-1ff43888ccd0\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " Nov 03 22:08:36.971693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.971616 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-oauth-config\") pod \"09683956-f309-4c54-b6cb-1ff43888ccd0\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " Nov 03 22:08:36.971693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.971669 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-service-ca\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-service-ca\") pod \"09683956-f309-4c54-b6cb-1ff43888ccd0\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " Nov 03 22:08:36.971911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.971722 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-oauth-serving-cert\") pod \"09683956-f309-4c54-b6cb-1ff43888ccd0\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " Nov 03 22:08:36.971911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.971765 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09683956-f309-4c54-b6cb-1ff43888ccd0-kube-api-access-bqwfj\" (UniqueName: \"kubernetes.io/projected/09683956-f309-4c54-b6cb-1ff43888ccd0-kube-api-access-bqwfj\") pod \"09683956-f309-4c54-b6cb-1ff43888ccd0\" (UID: \"09683956-f309-4c54-b6cb-1ff43888ccd0\") " Nov 03 22:08:36.971911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.971892 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-console-config" pod "09683956-f309-4c54-b6cb-1ff43888ccd0" (UID: "09683956-f309-4c54-b6cb-1ff43888ccd0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:36.972051 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.972033 4298 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-console-config\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:36.972568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.972483 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-service-ca" pod "09683956-f309-4c54-b6cb-1ff43888ccd0" (UID: "09683956-f309-4c54-b6cb-1ff43888ccd0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:36.972727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.972583 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09683956-f309-4c54-b6cb-1ff43888ccd0-tmp" pod "09683956-f309-4c54-b6cb-1ff43888ccd0" (UID: "09683956-f309-4c54-b6cb-1ff43888ccd0"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:36.973099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.973075 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-oauth-serving-cert" pod "09683956-f309-4c54-b6cb-1ff43888ccd0" (UID: "09683956-f309-4c54-b6cb-1ff43888ccd0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:36.974658 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.974622 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-oauth-config" pod "09683956-f309-4c54-b6cb-1ff43888ccd0" (UID: "09683956-f309-4c54-b6cb-1ff43888ccd0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:36.974884 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.974856 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-serving-cert" pod "09683956-f309-4c54-b6cb-1ff43888ccd0" (UID: "09683956-f309-4c54-b6cb-1ff43888ccd0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:36.974884 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:36.974875 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09683956-f309-4c54-b6cb-1ff43888ccd0-kube-api-access-bqwfj" pod "09683956-f309-4c54-b6cb-1ff43888ccd0" (UID: "09683956-f309-4c54-b6cb-1ff43888ccd0"). InnerVolumeSpecName "kube-api-access-bqwfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:37.073033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.072993 4298 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-serving-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:37.073033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.073024 4298 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09683956-f309-4c54-b6cb-1ff43888ccd0-console-oauth-config\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:37.073033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.073034 4298 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:37.073033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.073043 4298 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09683956-f309-4c54-b6cb-1ff43888ccd0-oauth-serving-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:37.073287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.073052 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqwfj\" (UniqueName: \"kubernetes.io/projected/09683956-f309-4c54-b6cb-1ff43888ccd0-kube-api-access-bqwfj\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:37.073287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.073062 4298 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09683956-f309-4c54-b6cb-1ff43888ccd0-tmp\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:37.509689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.509638 4298 generic.go:356] "Generic (PLEG): container finished" podID="09683956-f309-4c54-b6cb-1ff43888ccd0" containerID="787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff" exitCode=0 Nov 03 22:08:37.509848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.509705 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc6474b6-98qng" Nov 03 22:08:37.509848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.509727 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc6474b6-98qng" event={"ID":"09683956-f309-4c54-b6cb-1ff43888ccd0","Type":"ContainerDied","Data":"787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff"} Nov 03 22:08:37.509848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.509766 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc6474b6-98qng" event={"ID":"09683956-f309-4c54-b6cb-1ff43888ccd0","Type":"ContainerDied","Data":"75600bd178c8ec359987e35ef5ba31155180cfa985299a43362595e88be98ca4"} Nov 03 22:08:37.509848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.509785 4298 scope.go:122] "RemoveContainer" containerID="787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff" Nov 03 22:08:37.517637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.517618 4298 scope.go:122] "RemoveContainer" containerID="787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff" Nov 03 22:08:37.517933 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:08:37.517912 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff\": container with ID starting with 787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff not found: ID does not exist" containerID="787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff" Nov 03 22:08:37.518001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.517945 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff"} err="failed to get container status \"787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff\": rpc error: code = NotFound desc = could not find container \"787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff\": container with ID starting with 787fe53043930c9d03629bb6f7dd9b9e30703ac3d98b8ac575f89f54b542c8ff not found: ID does not exist" Nov 03 22:08:37.676019 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.675987 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc6474b6-98qng"] Nov 03 22:08:37.726942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:37.726912 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bc6474b6-98qng"] Nov 03 22:08:39.132434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:39.132399 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09683956-f309-4c54-b6cb-1ff43888ccd0" path="/var/lib/kubelet/pods/09683956-f309-4c54-b6cb-1ff43888ccd0/volumes" Nov 03 22:08:44.236841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:44.236809 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:44.411843 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:44.411809 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:45.768515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:45.768478 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:46.166298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:46.166218 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:47.551056 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:08:47.551016 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:08:56.115259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:56.115223 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74d55f589f-dwsmd"] Nov 03 22:08:56.835243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:56.835202 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:57.017768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:08:57.017738 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:02.872078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:02.872039 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:11.877677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:11.877625 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:12.049781 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:12.049749 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:17.551181 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:09:17.551135 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:09:21.134533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.134480 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-74d55f589f-dwsmd" podUID="0a05ce10-05cb-433d-aa03-b6e9a4190eac" containerName="console" containerID="cri-o://eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0" gracePeriod=15 Nov 03 22:09:21.394764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.394706 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:09:21.513229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513196 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-serving-cert\") pod \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " Nov 03 22:09:21.513229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513237 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-oauth-config\") pod \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " Nov 03 22:09:21.513473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513259 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-trusted-ca-bundle\") pod \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " Nov 03 22:09:21.513473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513343 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-config\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-config\") pod \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " Nov 03 22:09:21.513473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513381 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a05ce10-05cb-433d-aa03-b6e9a4190eac-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a05ce10-05cb-433d-aa03-b6e9a4190eac-tmp\") pod \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " Nov 03 22:09:21.513473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513428 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a05ce10-05cb-433d-aa03-b6e9a4190eac-kube-api-access-dz5fk\" (UniqueName: \"kubernetes.io/projected/0a05ce10-05cb-433d-aa03-b6e9a4190eac-kube-api-access-dz5fk\") pod \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " Nov 03 22:09:21.513697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513510 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-service-ca\") pod \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " Nov 03 22:09:21.513697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513548 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-oauth-serving-cert\") pod \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\" (UID: \"0a05ce10-05cb-433d-aa03-b6e9a4190eac\") " Nov 03 22:09:21.513697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513677 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a05ce10-05cb-433d-aa03-b6e9a4190eac-tmp" pod "0a05ce10-05cb-433d-aa03-b6e9a4190eac" (UID: "0a05ce10-05cb-433d-aa03-b6e9a4190eac"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:21.513855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513787 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-trusted-ca-bundle" pod "0a05ce10-05cb-433d-aa03-b6e9a4190eac" (UID: "0a05ce10-05cb-433d-aa03-b6e9a4190eac"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:21.513919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513901 4298 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-trusted-ca-bundle\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:21.513979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513923 4298 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a05ce10-05cb-433d-aa03-b6e9a4190eac-tmp\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:21.513979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.513949 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-oauth-serving-cert" pod "0a05ce10-05cb-433d-aa03-b6e9a4190eac" (UID: "0a05ce10-05cb-433d-aa03-b6e9a4190eac"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:21.514135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.514116 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-service-ca" pod "0a05ce10-05cb-433d-aa03-b6e9a4190eac" (UID: "0a05ce10-05cb-433d-aa03-b6e9a4190eac"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:21.514383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.514357 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-config" pod "0a05ce10-05cb-433d-aa03-b6e9a4190eac" (UID: "0a05ce10-05cb-433d-aa03-b6e9a4190eac"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:21.515572 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.515543 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-serving-cert" pod "0a05ce10-05cb-433d-aa03-b6e9a4190eac" (UID: "0a05ce10-05cb-433d-aa03-b6e9a4190eac"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:09:21.515709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.515653 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-oauth-config" pod "0a05ce10-05cb-433d-aa03-b6e9a4190eac" (UID: "0a05ce10-05cb-433d-aa03-b6e9a4190eac"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:09:21.516162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.516140 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a05ce10-05cb-433d-aa03-b6e9a4190eac-kube-api-access-dz5fk" pod "0a05ce10-05cb-433d-aa03-b6e9a4190eac" (UID: "0a05ce10-05cb-433d-aa03-b6e9a4190eac"). InnerVolumeSpecName "kube-api-access-dz5fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:21.614737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.614697 4298 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:21.614737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.614737 4298 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-oauth-serving-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:21.614964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.614749 4298 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-serving-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:21.614964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.614760 4298 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-oauth-config\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:21.614964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.614773 4298 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a05ce10-05cb-433d-aa03-b6e9a4190eac-console-config\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:21.614964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.614788 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz5fk\" (UniqueName: \"kubernetes.io/projected/0a05ce10-05cb-433d-aa03-b6e9a4190eac-kube-api-access-dz5fk\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:21.655638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.655559 4298 generic.go:356] "Generic (PLEG): container finished" podID="0a05ce10-05cb-433d-aa03-b6e9a4190eac" containerID="eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0" exitCode=0 Nov 03 22:09:21.655638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.655620 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d55f589f-dwsmd" Nov 03 22:09:21.655815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.655664 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d55f589f-dwsmd" event={"ID":"0a05ce10-05cb-433d-aa03-b6e9a4190eac","Type":"ContainerDied","Data":"eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0"} Nov 03 22:09:21.655815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.655715 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d55f589f-dwsmd" event={"ID":"0a05ce10-05cb-433d-aa03-b6e9a4190eac","Type":"ContainerDied","Data":"209bf9cf83d911aa33a534460f2b68d565ed6704e8733f74bfd3c46cf675d9c3"} Nov 03 22:09:21.655815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.655737 4298 scope.go:122] "RemoveContainer" containerID="eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0" Nov 03 22:09:21.664696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.664679 4298 scope.go:122] "RemoveContainer" containerID="eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0" Nov 03 22:09:21.664962 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:09:21.664945 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0\": container with ID starting with eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0 not found: ID does not exist" containerID="eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0" Nov 03 22:09:21.665005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.664972 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0"} err="failed to get container status \"eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0\": rpc error: code = NotFound desc = could not find container \"eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0\": container with ID starting with eea72362ab57e62cc5004cc080fa46e17983873d0253b54f34cede6e6020dfe0 not found: ID does not exist" Nov 03 22:09:21.822587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.822549 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74d55f589f-dwsmd"] Nov 03 22:09:21.825007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:21.824987 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74d55f589f-dwsmd"] Nov 03 22:09:23.131546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:09:23.131512 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a05ce10-05cb-433d-aa03-b6e9a4190eac" path="/var/lib/kubelet/pods/0a05ce10-05cb-433d-aa03-b6e9a4190eac/volumes" Nov 03 22:09:47.550716 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:09:47.550616 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:10:03.211247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:10:03.211207 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:03.445559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:10:03.445529 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:17.550276 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:10:17.550234 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:10:47.551097 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:10:47.551056 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:11:03.553071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:03.552990 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:17.552033 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:11:17.551997 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:11:26.669983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:26.669950 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:26.888103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:26.888072 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:28.075250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:28.075219 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:28.257958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:28.257928 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:29.085699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:29.085666 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:35.709829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:35.709795 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:36.526853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.526809 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7"] Nov 03 22:11:36.527206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.527188 4298 cpu_manager.go:414] "containerMap: removing container" podUID="0a05ce10-05cb-433d-aa03-b6e9a4190eac" containerName="console" Nov 03 22:11:36.527290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.527209 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a05ce10-05cb-433d-aa03-b6e9a4190eac" containerName="console" Nov 03 22:11:36.527290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.527235 4298 cpu_manager.go:414] "containerMap: removing container" podUID="09683956-f309-4c54-b6cb-1ff43888ccd0" containerName="console" Nov 03 22:11:36.527290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.527243 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09683956-f309-4c54-b6cb-1ff43888ccd0" containerName="console" Nov 03 22:11:36.527429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.527311 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a05ce10-05cb-433d-aa03-b6e9a4190eac" containerName="console" Nov 03 22:11:36.527429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.527326 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="09683956-f309-4c54-b6cb-1ff43888ccd0" containerName="console" Nov 03 22:11:36.530142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.530122 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.548590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.548566 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317f4b1e-43be-46a0-861e-f72df5c135ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.548723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.548600 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.548723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.548616 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.548723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.548631 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ww9p\" (UniqueName: \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-kube-api-access-9ww9p\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.548913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.548767 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.548913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.548816 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.548913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.548885 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.549008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.548936 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.549041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.549013 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.583658 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.583620 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:11:36.583658 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.583637 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:11:36.611510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.611455 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:11:36.611510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.611500 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kf4rv\"" Nov 03 22:11:36.636134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.636077 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7"] Nov 03 22:11:36.649375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.649346 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317f4b1e-43be-46a0-861e-f72df5c135ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.649375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.649379 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.649613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.649395 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.649613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.649413 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ww9p\" (UniqueName: \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-kube-api-access-9ww9p\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.649613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.649532 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.649613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.649579 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.649863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.649616 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.649863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.649681 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.649863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.649770 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.649863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.649853 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.650053 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.650019 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.650290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.650254 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317f4b1e-43be-46a0-861e-f72df5c135ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.650387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.650303 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.650387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.650340 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.652034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.652010 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.652139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.652047 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.707196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.707157 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.707342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.707253 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ww9p\" (UniqueName: \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-kube-api-access-9ww9p\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-jc9l7\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.831924 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.831847 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s"] Nov 03 22:11:36.836096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.836076 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:36.842350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.842320 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:36.889585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.889555 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-ng8jg\"" Nov 03 22:11:36.891268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.891203 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s"] Nov 03 22:11:36.951268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.951237 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:36.951452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.951275 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1eeca1b-876a-431b-b98c-eea04180c345-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:36.951452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.951299 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1eeca1b-876a-431b-b98c-eea04180c345-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:36.951452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.951371 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:36.951452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.951432 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:36.951638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.951528 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1eeca1b-876a-431b-b98c-eea04180c345-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:36.951638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.951556 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:36.951638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.951594 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:36.951778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:36.951700 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr765\" (UniqueName: \"kubernetes.io/projected/d1eeca1b-876a-431b-b98c-eea04180c345-kube-api-access-pr765\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.052341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052300 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.052341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052350 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.052571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052371 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1eeca1b-876a-431b-b98c-eea04180c345-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.052571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052390 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.052571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052422 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.052571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052443 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr765\" (UniqueName: \"kubernetes.io/projected/d1eeca1b-876a-431b-b98c-eea04180c345-kube-api-access-pr765\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.052571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052477 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.052571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052506 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1eeca1b-876a-431b-b98c-eea04180c345-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.052571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052538 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1eeca1b-876a-431b-b98c-eea04180c345-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.052931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052762 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.052931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052821 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.053009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.052938 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.053041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.053008 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.053245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.053215 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1eeca1b-876a-431b-b98c-eea04180c345-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.054944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.054926 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1eeca1b-876a-431b-b98c-eea04180c345-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.055142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.055121 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1eeca1b-876a-431b-b98c-eea04180c345-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.113598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.113531 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1eeca1b-876a-431b-b98c-eea04180c345-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.113739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.113697 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr765\" (UniqueName: \"kubernetes.io/projected/d1eeca1b-876a-431b-b98c-eea04180c345-kube-api-access-pr765\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s\" (UID: \"d1eeca1b-876a-431b-b98c-eea04180c345\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.149146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.149112 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:37.260737 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:11:37.260687 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317f4b1e_43be_46a0_861e_f72df5c135ce.slice/crio-ff3bff6cb0f4e7cbd685c7cf13978c5b5c2cb06cef2dff24f6dd54190c12e969 WatchSource:0}: Error finding container ff3bff6cb0f4e7cbd685c7cf13978c5b5c2cb06cef2dff24f6dd54190c12e969: Status 404 returned error can't find the container with id ff3bff6cb0f4e7cbd685c7cf13978c5b5c2cb06cef2dff24f6dd54190c12e969 Nov 03 22:11:37.261512 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.261452 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7"] Nov 03 22:11:37.514946 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:11:37.514917 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1eeca1b_876a_431b_b98c_eea04180c345.slice/crio-438e87712c7f03737016f66f63c2f658eab3f0c888e68d9ffb933bbd1851e629 WatchSource:0}: Error finding container 438e87712c7f03737016f66f63c2f658eab3f0c888e68d9ffb933bbd1851e629: Status 404 returned error can't find the container with id 438e87712c7f03737016f66f63c2f658eab3f0c888e68d9ffb933bbd1851e629 Nov 03 22:11:37.515460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.515435 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s"] Nov 03 22:11:37.632891 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:37.632857 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:38.110097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:38.110063 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" event={"ID":"d1eeca1b-876a-431b-b98c-eea04180c345","Type":"ContainerStarted","Data":"438e87712c7f03737016f66f63c2f658eab3f0c888e68d9ffb933bbd1851e629"} Nov 03 22:11:38.111210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:38.111187 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" event={"ID":"317f4b1e-43be-46a0-861e-f72df5c135ce","Type":"ContainerStarted","Data":"ff3bff6cb0f4e7cbd685c7cf13978c5b5c2cb06cef2dff24f6dd54190c12e969"} Nov 03 22:11:40.824563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:40.824528 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:41.454607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:41.454579 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:11:41.454705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:41.454665 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:11:41.454705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:41.454697 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:11:42.127298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:42.127254 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" event={"ID":"d1eeca1b-876a-431b-b98c-eea04180c345","Type":"ContainerStarted","Data":"5ef3404f2e8832c632e060ca041afe9f40349501c56439ad4d16c045e056fa21"} Nov 03 22:11:42.150227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:42.150200 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:42.154719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:42.154684 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:42.348945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:42.348884 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" podStartSLOduration=2.412162547 podStartE2EDuration="6.348869784s" totalImagesPullingTime="3.936707237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:42.297332529 +0000 UTC m=+259.278066182" watchObservedRunningTime="2026-11-03 22:11:42.348869784 +0000 UTC m=+259.329603436" Nov 03 22:11:43.131770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:43.131742 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:43.132153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:43.131933 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s" Nov 03 22:11:47.550385 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:11:47.550349 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:11:49.619597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:49.619566 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:11:49.619858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:49.619634 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:11:49.619858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:49.619683 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:11:50.156520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:50.156489 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" event={"ID":"317f4b1e-43be-46a0-861e-f72df5c135ce","Type":"ContainerStarted","Data":"ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb"} Nov 03 22:11:50.379177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:50.379120 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" podStartSLOduration=2.023006704 podStartE2EDuration="14.379103776s" totalImagesPullingTime="12.356097072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:50.328211481 +0000 UTC m=+267.308945138" watchObservedRunningTime="2026-11-03 22:11:50.379103776 +0000 UTC m=+267.359837429" Nov 03 22:11:50.842894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:50.842859 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:50.847270 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:50.847248 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:51.159482 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:51.159410 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:51.160356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:51.160340 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:11:58.725738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:58.725698 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:58.902684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:11:58.902627 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:00.076981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:00.076942 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:00.268427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:00.268394 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:01.748950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:01.748916 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:01.951859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:01.951827 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:03.905850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:03.905805 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:05.556753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:05.556720 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:05.834620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:05.834540 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:06.202474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.202445 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:06.402305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.402272 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb"] Nov 03 22:12:06.405702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.405678 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.453507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.453423 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb"] Nov 03 22:12:06.469702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.469676 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:06.508720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.508680 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca4b142d-366e-4801-8597-706f1160c4ad-metrics-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.508902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.508730 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ca4b142d-366e-4801-8597-706f1160c4ad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.508902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.508745 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca4b142d-366e-4801-8597-706f1160c4ad-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.508902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.508889 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca4b142d-366e-4801-8597-706f1160c4ad-tmp\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.509070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.508954 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwkn\" (UniqueName: \"kubernetes.io/projected/ca4b142d-366e-4801-8597-706f1160c4ad-kube-api-access-7rwkn\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.509070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.509009 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ca4b142d-366e-4801-8597-706f1160c4ad-manifests\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.550981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.550951 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 22:12:06.551175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.551018 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 22:12:06.551175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.551042 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 22:12:06.551175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.551095 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 22:12:06.551175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.551105 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 22:12:06.551175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.551116 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n7ktg\"" Nov 03 22:12:06.610331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.610285 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwkn\" (UniqueName: \"kubernetes.io/projected/ca4b142d-366e-4801-8597-706f1160c4ad-kube-api-access-7rwkn\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.610847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.610348 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ca4b142d-366e-4801-8597-706f1160c4ad-manifests\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.610847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.610402 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca4b142d-366e-4801-8597-706f1160c4ad-metrics-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.610847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.610434 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ca4b142d-366e-4801-8597-706f1160c4ad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.610847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.610620 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca4b142d-366e-4801-8597-706f1160c4ad-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.610847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.610714 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca4b142d-366e-4801-8597-706f1160c4ad-tmp\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.611166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.610895 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ca4b142d-366e-4801-8597-706f1160c4ad-manifests\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.612935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.612900 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca4b142d-366e-4801-8597-706f1160c4ad-tmp\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.613059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.612943 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca4b142d-366e-4801-8597-706f1160c4ad-metrics-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.613059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.613016 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca4b142d-366e-4801-8597-706f1160c4ad-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.613059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.613018 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ca4b142d-366e-4801-8597-706f1160c4ad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.698373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.698339 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwkn\" (UniqueName: \"kubernetes.io/projected/ca4b142d-366e-4801-8597-706f1160c4ad-kube-api-access-7rwkn\") pod \"opendatahub-operator-controller-manager-55b449f4f4-4p5nb\" (UID: \"ca4b142d-366e-4801-8597-706f1160c4ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.720397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.720301 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:06.807804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:06.807766 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:07.016578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:07.016497 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:07.196474 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:12:07.196442 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4b142d_366e_4801_8597_706f1160c4ad.slice/crio-ec2ceb647bf311238b389835a9df6ad8febd10a3a856c014bdab478d3a5db531 WatchSource:0}: Error finding container ec2ceb647bf311238b389835a9df6ad8febd10a3a856c014bdab478d3a5db531: Status 404 returned error can't find the container with id ec2ceb647bf311238b389835a9df6ad8febd10a3a856c014bdab478d3a5db531 Nov 03 22:12:07.198699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:07.198676 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb"] Nov 03 22:12:07.212515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:07.212488 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" event={"ID":"ca4b142d-366e-4801-8597-706f1160c4ad","Type":"ContainerStarted","Data":"ec2ceb647bf311238b389835a9df6ad8febd10a3a856c014bdab478d3a5db531"} Nov 03 22:12:07.426858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:07.426780 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:07.633997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:07.633967 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:09.216560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:09.216509 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:09.406540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:09.406429 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:10.784120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:10.784088 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:11.228883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:11.228849 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" event={"ID":"ca4b142d-366e-4801-8597-706f1160c4ad","Type":"ContainerStarted","Data":"597492379c5a00c55e21a0824caf9f00788b82f73f7208cf2b169e8b405dfbd9"} Nov 03 22:12:12.234431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:12.234397 4298 generic.go:356] "Generic (PLEG): container finished" podID="ca4b142d-366e-4801-8597-706f1160c4ad" containerID="597492379c5a00c55e21a0824caf9f00788b82f73f7208cf2b169e8b405dfbd9" exitCode=0 Nov 03 22:12:12.234879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:12.234485 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" event={"ID":"ca4b142d-366e-4801-8597-706f1160c4ad","Type":"ContainerDied","Data":"597492379c5a00c55e21a0824caf9f00788b82f73f7208cf2b169e8b405dfbd9"} Nov 03 22:12:13.244412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:13.244370 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" event={"ID":"ca4b142d-366e-4801-8597-706f1160c4ad","Type":"ContainerStarted","Data":"0170eab387af5dd918f1e7e8981f753f6555aabe56d6f12b4069e20579b40225"} Nov 03 22:12:13.244933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:13.244553 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:13.431577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:13.431476 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" podStartSLOduration=2.397931145 podStartE2EDuration="7.431463344s" totalImagesPullingTime="4.496342797s" totalInitContainerRuntime="537.189402ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:13.429818845 +0000 UTC m=+290.410552498" watchObservedRunningTime="2026-11-03 22:12:13.431463344 +0000 UTC m=+290.412197065" Nov 03 22:12:14.390031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:14.389985 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:14.579205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:14.579172 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:17.551452 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:12:17.551415 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:12:18.162728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:18.162694 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:18.359973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:18.359940 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:19.663296 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:19.663260 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:21.684757 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:21.684723 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:21.932237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:21.932208 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:22.129233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:22.129197 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:22.763170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:22.763135 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:23.904662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:23.904583 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:24.250304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:24.250271 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:12:24.360523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:24.360487 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:24.778895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:24.778859 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:26.185738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:26.185704 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:27.006619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:27.006564 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:27.394874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:27.394845 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:27.610894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:27.610861 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:27.851715 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:27.851683 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:28.045205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:28.045176 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:28.976851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:28.976805 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:29.728126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:29.728093 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh"] Nov 03 22:12:29.734046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:29.734022 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh" Nov 03 22:12:29.784615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:29.784582 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh"] Nov 03 22:12:29.834725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:29.834697 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 22:12:29.834878 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:29.834748 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 22:12:29.847113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:29.847080 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fvv9p\"" Nov 03 22:12:29.904673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:29.904626 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqp8\" (UniqueName: \"kubernetes.io/projected/de8fee81-906b-4f2c-9c9c-5b5303095361-kube-api-access-vvqp8\") pod \"cert-manager-cainjector-5c57bbd4fc-tb4xh\" (UID: \"de8fee81-906b-4f2c-9c9c-5b5303095361\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh" Nov 03 22:12:29.904834 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:29.904719 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de8fee81-906b-4f2c-9c9c-5b5303095361-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tb4xh\" (UID: \"de8fee81-906b-4f2c-9c9c-5b5303095361\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh" Nov 03 22:12:30.006208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:30.006114 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqp8\" (UniqueName: \"kubernetes.io/projected/de8fee81-906b-4f2c-9c9c-5b5303095361-kube-api-access-vvqp8\") pod \"cert-manager-cainjector-5c57bbd4fc-tb4xh\" (UID: \"de8fee81-906b-4f2c-9c9c-5b5303095361\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh" Nov 03 22:12:30.006208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:30.006162 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de8fee81-906b-4f2c-9c9c-5b5303095361-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tb4xh\" (UID: \"de8fee81-906b-4f2c-9c9c-5b5303095361\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh" Nov 03 22:12:30.064405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:30.064375 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de8fee81-906b-4f2c-9c9c-5b5303095361-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tb4xh\" (UID: \"de8fee81-906b-4f2c-9c9c-5b5303095361\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh" Nov 03 22:12:30.064566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:30.064537 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqp8\" (UniqueName: \"kubernetes.io/projected/de8fee81-906b-4f2c-9c9c-5b5303095361-kube-api-access-vvqp8\") pod \"cert-manager-cainjector-5c57bbd4fc-tb4xh\" (UID: \"de8fee81-906b-4f2c-9c9c-5b5303095361\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh" Nov 03 22:12:30.353215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:30.353135 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh" Nov 03 22:12:30.715365 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:12:30.715336 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde8fee81_906b_4f2c_9c9c_5b5303095361.slice/crio-b254ffb822af4101ac4df8f128dfcd35b55867d2157eb16aae3e459fc1c8e64f WatchSource:0}: Error finding container b254ffb822af4101ac4df8f128dfcd35b55867d2157eb16aae3e459fc1c8e64f: Status 404 returned error can't find the container with id b254ffb822af4101ac4df8f128dfcd35b55867d2157eb16aae3e459fc1c8e64f Nov 03 22:12:30.716492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:30.716471 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh"] Nov 03 22:12:30.717224 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:30.717208 4298 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:12:31.223502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:31.223471 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:31.305114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:31.305082 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh" event={"ID":"de8fee81-906b-4f2c-9c9c-5b5303095361","Type":"ContainerStarted","Data":"b254ffb822af4101ac4df8f128dfcd35b55867d2157eb16aae3e459fc1c8e64f"} Nov 03 22:12:31.441269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:31.441224 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:32.045058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.045025 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:32.258632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.258597 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-rldrx"] Nov 03 22:12:32.263211 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.263189 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rldrx" Nov 03 22:12:32.315212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.315145 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rldrx"] Nov 03 22:12:32.324530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.324498 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fttdr\" (UniqueName: \"kubernetes.io/projected/f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3-kube-api-access-fttdr\") pod \"cert-manager-597b95cb84-rldrx\" (UID: \"f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3\") " pod="cert-manager/cert-manager-597b95cb84-rldrx" Nov 03 22:12:32.324676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.324573 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3-bound-sa-token\") pod \"cert-manager-597b95cb84-rldrx\" (UID: \"f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3\") " pod="cert-manager/cert-manager-597b95cb84-rldrx" Nov 03 22:12:32.365234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.365204 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-28sxb\"" Nov 03 22:12:32.425360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.425326 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fttdr\" (UniqueName: \"kubernetes.io/projected/f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3-kube-api-access-fttdr\") pod \"cert-manager-597b95cb84-rldrx\" (UID: \"f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3\") " pod="cert-manager/cert-manager-597b95cb84-rldrx" Nov 03 22:12:32.425531 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.425417 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3-bound-sa-token\") pod \"cert-manager-597b95cb84-rldrx\" (UID: \"f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3\") " pod="cert-manager/cert-manager-597b95cb84-rldrx" Nov 03 22:12:32.488443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.488410 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3-bound-sa-token\") pod \"cert-manager-597b95cb84-rldrx\" (UID: \"f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3\") " pod="cert-manager/cert-manager-597b95cb84-rldrx" Nov 03 22:12:32.488591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.488468 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fttdr\" (UniqueName: \"kubernetes.io/projected/f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3-kube-api-access-fttdr\") pod \"cert-manager-597b95cb84-rldrx\" (UID: \"f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3\") " pod="cert-manager/cert-manager-597b95cb84-rldrx" Nov 03 22:12:32.574998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.574910 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rldrx" Nov 03 22:12:32.942120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:32.942094 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rldrx"] Nov 03 22:12:33.315045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:33.314999 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rldrx" event={"ID":"f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3","Type":"ContainerStarted","Data":"2814a89e963bc04c4361f6b0f9c2d470cf1a3dd8ca47560b6f4c25b2f937cee1"} Nov 03 22:12:35.352535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:35.352486 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:36.329988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:36.329946 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh" event={"ID":"de8fee81-906b-4f2c-9c9c-5b5303095361","Type":"ContainerStarted","Data":"8538d8269833c33b9bf4c23c747ad2017803817ec50527cc3ee04863dac35f0c"} Nov 03 22:12:36.331816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:36.331787 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rldrx" event={"ID":"f3b7ccd0-55e5-4db1-8ee4-526a21f8d0f3","Type":"ContainerStarted","Data":"9d5a0b3ba374ca7fb9beb50a6b20142666dde58e75024d0b86b0c95fbc48407b"} Nov 03 22:12:36.566805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:36.566752 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tb4xh" podStartSLOduration=2.551321633 podStartE2EDuration="7.566738755s" totalImagesPullingTime="5.015417122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:36.448159289 +0000 UTC m=+313.428892941" watchObservedRunningTime="2026-11-03 22:12:36.566738755 +0000 UTC m=+313.547472407" Nov 03 22:12:36.567238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:36.567205 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-rldrx" podStartSLOduration=1.625397276 podStartE2EDuration="4.567195087s" totalImagesPullingTime="2.941797811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:36.566080048 +0000 UTC m=+313.546813700" watchObservedRunningTime="2026-11-03 22:12:36.567195087 +0000 UTC m=+313.547928743" Nov 03 22:12:36.989917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:36.989869 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:37.636582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:37.636549 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl"] Nov 03 22:12:37.639988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:37.639971 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl" Nov 03 22:12:37.679971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:37.679930 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gq7d\" (UniqueName: \"kubernetes.io/projected/26112fa6-98ae-46b6-b961-292291a69e55-kube-api-access-6gq7d\") pod \"limitador-operator-controller-manager-5778bb4449-fg9bl\" (UID: \"26112fa6-98ae-46b6-b961-292291a69e55\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl" Nov 03 22:12:37.694276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:37.694246 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl"] Nov 03 22:12:37.697168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:37.697147 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 22:12:37.697263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:37.697185 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 22:12:37.743677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:37.743627 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-htjcj\"" Nov 03 22:12:37.780942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:37.780904 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gq7d\" (UniqueName: \"kubernetes.io/projected/26112fa6-98ae-46b6-b961-292291a69e55-kube-api-access-6gq7d\") pod \"limitador-operator-controller-manager-5778bb4449-fg9bl\" (UID: \"26112fa6-98ae-46b6-b961-292291a69e55\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl" Nov 03 22:12:37.841221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:37.841187 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gq7d\" (UniqueName: \"kubernetes.io/projected/26112fa6-98ae-46b6-b961-292291a69e55-kube-api-access-6gq7d\") pod \"limitador-operator-controller-manager-5778bb4449-fg9bl\" (UID: \"26112fa6-98ae-46b6-b961-292291a69e55\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl" Nov 03 22:12:37.952020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:37.951988 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl" Nov 03 22:12:38.375502 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:12:38.375458 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26112fa6_98ae_46b6_b961_292291a69e55.slice/crio-7e88427422fea7f1a528d8b762e0b88614ca01d23cb6e8e7530cae2843631e54 WatchSource:0}: Error finding container 7e88427422fea7f1a528d8b762e0b88614ca01d23cb6e8e7530cae2843631e54: Status 404 returned error can't find the container with id 7e88427422fea7f1a528d8b762e0b88614ca01d23cb6e8e7530cae2843631e54 Nov 03 22:12:38.377295 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:38.377268 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl"] Nov 03 22:12:38.989157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:38.989118 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:39.208450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:39.208407 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:39.345735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:39.345634 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl" event={"ID":"26112fa6-98ae-46b6-b961-292291a69e55","Type":"ContainerStarted","Data":"7e88427422fea7f1a528d8b762e0b88614ca01d23cb6e8e7530cae2843631e54"} Nov 03 22:12:39.609554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:39.609472 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:39.828199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:39.828164 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:40.913638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:40.913606 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:41.152930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.152892 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:41.616226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.616192 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn"] Nov 03 22:12:41.624743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.624719 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.653695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.653662 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:41.676022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.675989 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn"] Nov 03 22:12:41.716922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.716873 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.717102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.716938 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.717102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.717022 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.717102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.717063 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76c20eb-1d4b-408a-a86f-38aa1198c340-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.717215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.717125 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.717215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.717143 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6l6\" (UniqueName: \"kubernetes.io/projected/b76c20eb-1d4b-408a-a86f-38aa1198c340-kube-api-access-2s6l6\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.717215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.717171 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.717215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.717197 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-token\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.717349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.717214 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-data\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.725452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.725426 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hfgqt\"" Nov 03 22:12:41.817708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.817674 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.817708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.817720 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76c20eb-1d4b-408a-a86f-38aa1198c340-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.817934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.817769 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.817934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.817795 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6l6\" (UniqueName: \"kubernetes.io/projected/b76c20eb-1d4b-408a-a86f-38aa1198c340-kube-api-access-2s6l6\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.817934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.817827 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.817934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.817865 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-token\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.817934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.817893 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-data\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.817934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.817932 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.818259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.817994 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.818259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.818097 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.818639 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.818498 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76c20eb-1d4b-408a-a86f-38aa1198c340-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.818639 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.818607 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.818639 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.818689 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-data\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.818639 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.818737 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.821063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.821030 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.821248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.821229 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.848497 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.848458 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:41.879309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.879240 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76c20eb-1d4b-408a-a86f-38aa1198c340-istio-token\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.882449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.882419 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6l6\" (UniqueName: \"kubernetes.io/projected/b76c20eb-1d4b-408a-a86f-38aa1198c340-kube-api-access-2s6l6\") pod \"data-science-gateway-data-science-gateway-class-597d745469jn4qn\" (UID: \"b76c20eb-1d4b-408a-a86f-38aa1198c340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:41.938540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:41.938505 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:42.049325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:42.049283 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:42.364623 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:42.364593 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn"] Nov 03 22:12:42.396602 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:12:42.396577 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76c20eb_1d4b_408a_a86f_38aa1198c340.slice/crio-e6a76aa911eb404bbaf7b0f61cf84e8352402e5500a9571eaab887d5e862ada9 WatchSource:0}: Error finding container e6a76aa911eb404bbaf7b0f61cf84e8352402e5500a9571eaab887d5e862ada9: Status 404 returned error can't find the container with id e6a76aa911eb404bbaf7b0f61cf84e8352402e5500a9571eaab887d5e862ada9 Nov 03 22:12:42.398568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:42.398538 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:12:42.398632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:42.398610 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:12:42.398689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:42.398654 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:12:43.364744 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:43.364710 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl" event={"ID":"26112fa6-98ae-46b6-b961-292291a69e55","Type":"ContainerStarted","Data":"42c7f3701e5ec9ff517a08ffdd305860c1c5e09bba4c3fbd8883df85329578cd"} Nov 03 22:12:43.365175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:43.364827 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl" Nov 03 22:12:43.366232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:43.366211 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" event={"ID":"b76c20eb-1d4b-408a-a86f-38aa1198c340","Type":"ContainerStarted","Data":"714e4b2b5a06eb67b882aceed8dc24b432793e9b3986c364e5ed911e6c788c7c"} Nov 03 22:12:43.366324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:43.366246 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" event={"ID":"b76c20eb-1d4b-408a-a86f-38aa1198c340","Type":"ContainerStarted","Data":"e6a76aa911eb404bbaf7b0f61cf84e8352402e5500a9571eaab887d5e862ada9"} Nov 03 22:12:43.481926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:43.481866 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl" podStartSLOduration=2.5407638930000003 podStartE2EDuration="6.481851611s" totalImagesPullingTime="3.941087718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:43.481162058 +0000 UTC m=+320.461895708" watchObservedRunningTime="2026-11-03 22:12:43.481851611 +0000 UTC m=+320.462585264" Nov 03 22:12:43.601571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:43.601522 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" podStartSLOduration=2.601508196 podStartE2EDuration="2.601508196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:43.601215187 +0000 UTC m=+320.581948840" watchObservedRunningTime="2026-11-03 22:12:43.601508196 +0000 UTC m=+320.582241849" Nov 03 22:12:43.938764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:43.938727 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:43.943621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:43.943595 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:44.369891 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:44.369853 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:44.370696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:44.370677 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469jn4qn" Nov 03 22:12:44.763753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:44.763718 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:44.963796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:44.963765 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:45.163774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:45.163700 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:46.001497 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.001457 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:46.021835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.021802 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6gbn"] Nov 03 22:12:46.029942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.029920 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:12:46.081797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.081763 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6gbn"] Nov 03 22:12:46.130902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.130866 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 22:12:46.131104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.130923 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mhkbs\"" Nov 03 22:12:46.158464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.158425 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e082092-2d53-4edc-8a30-1a412abb8423-config-file\") pod \"limitador-limitador-74dfcfb894-d6gbn\" (UID: \"9e082092-2d53-4edc-8a30-1a412abb8423\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:12:46.158629 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.158466 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xxd\" (UniqueName: \"kubernetes.io/projected/9e082092-2d53-4edc-8a30-1a412abb8423-kube-api-access-k9xxd\") pod \"limitador-limitador-74dfcfb894-d6gbn\" (UID: \"9e082092-2d53-4edc-8a30-1a412abb8423\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:12:46.259384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.259278 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e082092-2d53-4edc-8a30-1a412abb8423-config-file\") pod \"limitador-limitador-74dfcfb894-d6gbn\" (UID: \"9e082092-2d53-4edc-8a30-1a412abb8423\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:12:46.259384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.259312 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xxd\" (UniqueName: \"kubernetes.io/projected/9e082092-2d53-4edc-8a30-1a412abb8423-kube-api-access-k9xxd\") pod \"limitador-limitador-74dfcfb894-d6gbn\" (UID: \"9e082092-2d53-4edc-8a30-1a412abb8423\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:12:46.259966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.259945 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e082092-2d53-4edc-8a30-1a412abb8423-config-file\") pod \"limitador-limitador-74dfcfb894-d6gbn\" (UID: \"9e082092-2d53-4edc-8a30-1a412abb8423\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:12:46.274824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.274790 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6gbn"] Nov 03 22:12:46.317256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.317225 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xxd\" (UniqueName: \"kubernetes.io/projected/9e082092-2d53-4edc-8a30-1a412abb8423-kube-api-access-k9xxd\") pod \"limitador-limitador-74dfcfb894-d6gbn\" (UID: \"9e082092-2d53-4edc-8a30-1a412abb8423\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:12:46.338788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.338761 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:46.342753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.342730 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:12:46.563620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.563542 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:46.714845 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:12:46.714811 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e082092_2d53_4edc_8a30_1a412abb8423.slice/crio-58ee544b125ffb9ee603081acbea648c0e7ec7466811e422c4eda3bb72a4be9c WatchSource:0}: Error finding container 58ee544b125ffb9ee603081acbea648c0e7ec7466811e422c4eda3bb72a4be9c: Status 404 returned error can't find the container with id 58ee544b125ffb9ee603081acbea648c0e7ec7466811e422c4eda3bb72a4be9c Nov 03 22:12:46.716389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:46.716364 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6gbn"] Nov 03 22:12:47.382984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:47.382949 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" event={"ID":"9e082092-2d53-4edc-8a30-1a412abb8423","Type":"ContainerStarted","Data":"58ee544b125ffb9ee603081acbea648c0e7ec7466811e422c4eda3bb72a4be9c"} Nov 03 22:12:47.551734 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:12:47.551696 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:12:47.705366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:47.705335 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:47.880351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:47.880317 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:48.964993 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:48.964961 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:49.915380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:49.915352 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:50.106521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.106491 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:50.505197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.505168 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ml5tf"] Nov 03 22:12:50.509772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.509748 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" Nov 03 22:12:50.576343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.576317 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vmmlk\"" Nov 03 22:12:50.596494 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.596461 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnzhl\" (UniqueName: \"kubernetes.io/projected/604d18f3-b84d-45e6-a840-61f673b79eb7-kube-api-access-jnzhl\") pod \"authorino-56c6c9ddd5-ml5tf\" (UID: \"604d18f3-b84d-45e6-a840-61f673b79eb7\") " pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" Nov 03 22:12:50.628293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.628263 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ml5tf"] Nov 03 22:12:50.695238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.695205 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5"] Nov 03 22:12:50.697268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.697245 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnzhl\" (UniqueName: \"kubernetes.io/projected/604d18f3-b84d-45e6-a840-61f673b79eb7-kube-api-access-jnzhl\") pod \"authorino-56c6c9ddd5-ml5tf\" (UID: \"604d18f3-b84d-45e6-a840-61f673b79eb7\") " pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" Nov 03 22:12:50.698512 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.698498 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" Nov 03 22:12:50.716945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.716913 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:50.751299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.751256 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5"] Nov 03 22:12:50.751979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.751950 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 22:12:50.751979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.751948 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 22:12:50.756240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.756170 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnzhl\" (UniqueName: \"kubernetes.io/projected/604d18f3-b84d-45e6-a840-61f673b79eb7-kube-api-access-jnzhl\") pod \"authorino-56c6c9ddd5-ml5tf\" (UID: \"604d18f3-b84d-45e6-a840-61f673b79eb7\") " pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" Nov 03 22:12:50.798571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.798540 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2kvf\" (UniqueName: \"kubernetes.io/projected/30487de4-4746-4395-befd-7cf626ee1bf8-kube-api-access-h2kvf\") pod \"kuadrant-console-plugin-66c4fb575d-75lm5\" (UID: \"30487de4-4746-4395-befd-7cf626ee1bf8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" Nov 03 22:12:50.798751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.798593 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/30487de4-4746-4395-befd-7cf626ee1bf8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-75lm5\" (UID: \"30487de4-4746-4395-befd-7cf626ee1bf8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" Nov 03 22:12:50.798751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.798685 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30487de4-4746-4395-befd-7cf626ee1bf8-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-75lm5\" (UID: \"30487de4-4746-4395-befd-7cf626ee1bf8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" Nov 03 22:12:50.821775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.821737 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" Nov 03 22:12:50.899239 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.899181 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2kvf\" (UniqueName: \"kubernetes.io/projected/30487de4-4746-4395-befd-7cf626ee1bf8-kube-api-access-h2kvf\") pod \"kuadrant-console-plugin-66c4fb575d-75lm5\" (UID: \"30487de4-4746-4395-befd-7cf626ee1bf8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" Nov 03 22:12:50.899395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.899251 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/30487de4-4746-4395-befd-7cf626ee1bf8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-75lm5\" (UID: \"30487de4-4746-4395-befd-7cf626ee1bf8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" Nov 03 22:12:50.899395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.899279 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30487de4-4746-4395-befd-7cf626ee1bf8-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-75lm5\" (UID: \"30487de4-4746-4395-befd-7cf626ee1bf8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" Nov 03 22:12:50.899990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.899967 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30487de4-4746-4395-befd-7cf626ee1bf8-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-75lm5\" (UID: \"30487de4-4746-4395-befd-7cf626ee1bf8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" Nov 03 22:12:50.901807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.901784 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/30487de4-4746-4395-befd-7cf626ee1bf8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-75lm5\" (UID: \"30487de4-4746-4395-befd-7cf626ee1bf8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" Nov 03 22:12:50.959114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.959084 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2kvf\" (UniqueName: \"kubernetes.io/projected/30487de4-4746-4395-befd-7cf626ee1bf8-kube-api-access-h2kvf\") pod \"kuadrant-console-plugin-66c4fb575d-75lm5\" (UID: \"30487de4-4746-4395-befd-7cf626ee1bf8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" Nov 03 22:12:50.980823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:50.980792 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:51.009935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:51.009860 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" Nov 03 22:12:51.185268 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:12:51.185232 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod604d18f3_b84d_45e6_a840_61f673b79eb7.slice/crio-1f26d2901a96c21b39c70ae7f239d2e1f57aba7b7ece09c29660400ab08c2f1d WatchSource:0}: Error finding container 1f26d2901a96c21b39c70ae7f239d2e1f57aba7b7ece09c29660400ab08c2f1d: Status 404 returned error can't find the container with id 1f26d2901a96c21b39c70ae7f239d2e1f57aba7b7ece09c29660400ab08c2f1d Nov 03 22:12:51.187018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:51.186997 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ml5tf"] Nov 03 22:12:51.374731 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:12:51.374704 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30487de4_4746_4395_befd_7cf626ee1bf8.slice/crio-2d27c08007af2cc959b0c620d1832fd76e9db1b6b6a017f61e3df95db7c3ef8c WatchSource:0}: Error finding container 2d27c08007af2cc959b0c620d1832fd76e9db1b6b6a017f61e3df95db7c3ef8c: Status 404 returned error can't find the container with id 2d27c08007af2cc959b0c620d1832fd76e9db1b6b6a017f61e3df95db7c3ef8c Nov 03 22:12:51.377078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:51.377056 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5"] Nov 03 22:12:51.399375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:51.399343 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" event={"ID":"604d18f3-b84d-45e6-a840-61f673b79eb7","Type":"ContainerStarted","Data":"1f26d2901a96c21b39c70ae7f239d2e1f57aba7b7ece09c29660400ab08c2f1d"} Nov 03 22:12:51.400358 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:51.400336 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" event={"ID":"30487de4-4746-4395-befd-7cf626ee1bf8","Type":"ContainerStarted","Data":"2d27c08007af2cc959b0c620d1832fd76e9db1b6b6a017f61e3df95db7c3ef8c"} Nov 03 22:12:52.405764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:52.405728 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" event={"ID":"9e082092-2d53-4edc-8a30-1a412abb8423","Type":"ContainerStarted","Data":"4b02835a1afd44cfd6c2a66f8e1a051f99c7a84eedfc255e5d57dd5facdf9e70"} Nov 03 22:12:52.406129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:52.405855 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:12:52.522505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:52.522449 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" podStartSLOduration=1.937685209 podStartE2EDuration="7.522434332s" totalImagesPullingTime="5.584749123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:52.521906231 +0000 UTC m=+329.502639886" watchObservedRunningTime="2026-11-03 22:12:52.522434332 +0000 UTC m=+329.503167985" Nov 03 22:12:54.371978 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:54.371940 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fg9bl" Nov 03 22:12:54.417056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:54.417013 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" event={"ID":"604d18f3-b84d-45e6-a840-61f673b79eb7","Type":"ContainerStarted","Data":"56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692"} Nov 03 22:12:54.626708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:54.626579 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" podStartSLOduration=2.028419795 podStartE2EDuration="4.626565923s" totalImagesPullingTime="2.598146128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:54.625141147 +0000 UTC m=+331.605874799" watchObservedRunningTime="2026-11-03 22:12:54.626565923 +0000 UTC m=+331.607299576" Nov 03 22:12:56.956716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:56.956671 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:57.137272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:57.137240 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:59.560055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:59.560009 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:59.746368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:59.746332 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:59.927327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:12:59.927249 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:00.121311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:00.121274 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:01.450776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:01.450742 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" event={"ID":"30487de4-4746-4395-befd-7cf626ee1bf8","Type":"ContainerStarted","Data":"0d26cf164c83a9e14b270ba2da6fcfccd48f2f6d51e9bb9ce23df1519b023b7b"} Nov 03 22:13:01.572388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:01.572331 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75lm5" podStartSLOduration=1.591966957 podStartE2EDuration="11.572308422s" totalImagesPullingTime="9.980341465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:01.571419334 +0000 UTC m=+338.552152984" watchObservedRunningTime="2026-11-03 22:13:01.572308422 +0000 UTC m=+338.553042075" Nov 03 22:13:03.414268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:03.414233 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:13:04.230594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:04.230561 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:04.491480 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:04.491405 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:04.951181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:04.951151 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:05.127220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:05.127190 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:05.355831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:05.355747 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:05.537399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:05.537360 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:06.022700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:06.022669 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6gbn"] Nov 03 22:13:06.022915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:06.022854 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" podUID="9e082092-2d53-4edc-8a30-1a412abb8423" containerName="limitador" containerID="cri-o://4b02835a1afd44cfd6c2a66f8e1a051f99c7a84eedfc255e5d57dd5facdf9e70" gracePeriod=30 Nov 03 22:13:06.474379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:06.474347 4298 generic.go:356] "Generic (PLEG): container finished" podID="9e082092-2d53-4edc-8a30-1a412abb8423" containerID="4b02835a1afd44cfd6c2a66f8e1a051f99c7a84eedfc255e5d57dd5facdf9e70" exitCode=0 Nov 03 22:13:06.474571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:06.474417 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" event={"ID":"9e082092-2d53-4edc-8a30-1a412abb8423","Type":"ContainerDied","Data":"4b02835a1afd44cfd6c2a66f8e1a051f99c7a84eedfc255e5d57dd5facdf9e70"} Nov 03 22:13:06.582091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:06.582033 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:13:06.748842 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:06.748809 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e082092-2d53-4edc-8a30-1a412abb8423-kube-api-access-k9xxd\" (UniqueName: \"kubernetes.io/projected/9e082092-2d53-4edc-8a30-1a412abb8423-kube-api-access-k9xxd\") pod \"9e082092-2d53-4edc-8a30-1a412abb8423\" (UID: \"9e082092-2d53-4edc-8a30-1a412abb8423\") " Nov 03 22:13:06.749021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:06.748866 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e082092-2d53-4edc-8a30-1a412abb8423-config-file\" (UniqueName: \"kubernetes.io/configmap/9e082092-2d53-4edc-8a30-1a412abb8423-config-file\") pod \"9e082092-2d53-4edc-8a30-1a412abb8423\" (UID: \"9e082092-2d53-4edc-8a30-1a412abb8423\") " Nov 03 22:13:06.749208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:06.749184 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e082092-2d53-4edc-8a30-1a412abb8423-config-file" pod "9e082092-2d53-4edc-8a30-1a412abb8423" (UID: "9e082092-2d53-4edc-8a30-1a412abb8423"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:06.751049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:06.751026 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e082092-2d53-4edc-8a30-1a412abb8423-kube-api-access-k9xxd" pod "9e082092-2d53-4edc-8a30-1a412abb8423" (UID: "9e082092-2d53-4edc-8a30-1a412abb8423"). InnerVolumeSpecName "kube-api-access-k9xxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:13:06.849905 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:06.849821 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9xxd\" (UniqueName: \"kubernetes.io/projected/9e082092-2d53-4edc-8a30-1a412abb8423-kube-api-access-k9xxd\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:13:06.849905 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:06.849854 4298 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e082092-2d53-4edc-8a30-1a412abb8423-config-file\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:13:07.479405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:07.479378 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" Nov 03 22:13:07.479554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:07.479369 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6gbn" event={"ID":"9e082092-2d53-4edc-8a30-1a412abb8423","Type":"ContainerDied","Data":"58ee544b125ffb9ee603081acbea648c0e7ec7466811e422c4eda3bb72a4be9c"} Nov 03 22:13:07.479554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:07.479510 4298 scope.go:122] "RemoveContainer" containerID="4b02835a1afd44cfd6c2a66f8e1a051f99c7a84eedfc255e5d57dd5facdf9e70" Nov 03 22:13:07.648045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:07.648010 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6gbn"] Nov 03 22:13:07.658096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:07.658068 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:07.698995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:07.698961 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6gbn"] Nov 03 22:13:09.132833 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:09.132803 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e082092-2d53-4edc-8a30-1a412abb8423" path="/var/lib/kubelet/pods/9e082092-2d53-4edc-8a30-1a412abb8423/volumes" Nov 03 22:13:09.565677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:09.565625 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:09.745738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:09.745708 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:11.891802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:11.891768 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:12.075728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.075692 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:12.510275 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.510241 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj"] Nov 03 22:13:12.510577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.510565 4298 cpu_manager.go:414] "containerMap: removing container" podUID="9e082092-2d53-4edc-8a30-1a412abb8423" containerName="limitador" Nov 03 22:13:12.510620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.510578 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e082092-2d53-4edc-8a30-1a412abb8423" containerName="limitador" Nov 03 22:13:12.510676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.510671 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e082092-2d53-4edc-8a30-1a412abb8423" containerName="limitador" Nov 03 22:13:12.516574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.516558 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" Nov 03 22:13:12.567332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.567301 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj"] Nov 03 22:13:12.617315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.617278 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:13:12.617315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.617286 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 22:13:12.617524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.617324 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fmrvm\"" Nov 03 22:13:12.617524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.617361 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:13:12.637168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.637136 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:12.703261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.703223 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnt8r\" (UniqueName: \"kubernetes.io/projected/8bd983da-1d6f-4af1-a83d-3b2262efbd62-kube-api-access-gnt8r\") pod \"postgres-7cdbf6dc44-s66dj\" (UID: \"8bd983da-1d6f-4af1-a83d-3b2262efbd62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" Nov 03 22:13:12.703460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.703281 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8bd983da-1d6f-4af1-a83d-3b2262efbd62-data\") pod \"postgres-7cdbf6dc44-s66dj\" (UID: \"8bd983da-1d6f-4af1-a83d-3b2262efbd62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" Nov 03 22:13:12.804818 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.804727 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnt8r\" (UniqueName: \"kubernetes.io/projected/8bd983da-1d6f-4af1-a83d-3b2262efbd62-kube-api-access-gnt8r\") pod \"postgres-7cdbf6dc44-s66dj\" (UID: \"8bd983da-1d6f-4af1-a83d-3b2262efbd62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" Nov 03 22:13:12.804818 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.804812 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8bd983da-1d6f-4af1-a83d-3b2262efbd62-data\") pod \"postgres-7cdbf6dc44-s66dj\" (UID: \"8bd983da-1d6f-4af1-a83d-3b2262efbd62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" Nov 03 22:13:12.805220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.805200 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8bd983da-1d6f-4af1-a83d-3b2262efbd62-data\") pod \"postgres-7cdbf6dc44-s66dj\" (UID: \"8bd983da-1d6f-4af1-a83d-3b2262efbd62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" Nov 03 22:13:12.863331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:12.863303 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnt8r\" (UniqueName: \"kubernetes.io/projected/8bd983da-1d6f-4af1-a83d-3b2262efbd62-kube-api-access-gnt8r\") pod \"postgres-7cdbf6dc44-s66dj\" (UID: \"8bd983da-1d6f-4af1-a83d-3b2262efbd62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" Nov 03 22:13:13.129951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:13.129867 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" Nov 03 22:13:13.499096 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:13:13.499063 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd983da_1d6f_4af1_a83d_3b2262efbd62.slice/crio-940d35ac72d5ed62c09d04ad2be86e224a9765bd9d9207155df5f829d250eef1 WatchSource:0}: Error finding container 940d35ac72d5ed62c09d04ad2be86e224a9765bd9d9207155df5f829d250eef1: Status 404 returned error can't find the container with id 940d35ac72d5ed62c09d04ad2be86e224a9765bd9d9207155df5f829d250eef1 Nov 03 22:13:13.500596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:13.500571 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj"] Nov 03 22:13:13.505620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:13.505598 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" event={"ID":"8bd983da-1d6f-4af1-a83d-3b2262efbd62","Type":"ContainerStarted","Data":"940d35ac72d5ed62c09d04ad2be86e224a9765bd9d9207155df5f829d250eef1"} Nov 03 22:13:15.583167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:15.583134 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:15.778793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:15.778762 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:16.589979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:16.589931 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:16.762003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:16.761932 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:17.553229 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:13:17.553194 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:13:20.539239 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:20.539149 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" event={"ID":"8bd983da-1d6f-4af1-a83d-3b2262efbd62","Type":"ContainerStarted","Data":"f4ccf72b923d843012a4570bc45347d58b6e69c4a81ed00f17337d4203085f4e"} Nov 03 22:13:20.539681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:20.539244 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" Nov 03 22:13:20.654804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:20.654743 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" podStartSLOduration=1.945039415 podStartE2EDuration="8.654730059s" totalImagesPullingTime="6.709690644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:20.653912191 +0000 UTC m=+357.634645840" watchObservedRunningTime="2026-11-03 22:13:20.654730059 +0000 UTC m=+357.635463712" Nov 03 22:13:26.572342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:26.572309 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-s66dj" Nov 03 22:13:33.457325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:33.457283 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:33.639687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:33.639628 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:33.825774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:33.825687 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:33.998676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:33.998610 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:38.786616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:38.786584 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:38.996942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:38.996912 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:41.745306 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.745266 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc"] Nov 03 22:13:41.750890 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.750865 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:41.802663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.802610 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc"] Nov 03 22:13:41.851987 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.851954 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 22:13:41.852170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.852013 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 22:13:41.852170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.852111 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 22:13:41.852170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.852041 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 22:13:41.852170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.852062 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 22:13:41.852170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.852041 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-wxs6m\"" Nov 03 22:13:41.858379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.858354 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08b579ef-92ab-4fa5-a153-6be536d77795-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:41.858515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.858399 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/08b579ef-92ab-4fa5-a153-6be536d77795-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:41.858515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.858424 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08b579ef-92ab-4fa5-a153-6be536d77795-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:41.858590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.858561 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9r9c\" (UniqueName: \"kubernetes.io/projected/08b579ef-92ab-4fa5-a153-6be536d77795-kube-api-access-r9r9c\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:41.959267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.959225 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/08b579ef-92ab-4fa5-a153-6be536d77795-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:41.959451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.959286 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08b579ef-92ab-4fa5-a153-6be536d77795-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:41.959451 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:13:41.959324 4298 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 22:13:41.959451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.959329 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9r9c\" (UniqueName: \"kubernetes.io/projected/08b579ef-92ab-4fa5-a153-6be536d77795-kube-api-access-r9r9c\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:41.959451 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:13:41.959429 4298 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08b579ef-92ab-4fa5-a153-6be536d77795-proxy-source podName:08b579ef-92ab-4fa5-a153-6be536d77795 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:42.459404365 +0000 UTC m=+379.440138015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/08b579ef-92ab-4fa5-a153-6be536d77795-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" (UID: "08b579ef-92ab-4fa5-a153-6be536d77795") : configmap "usage-tenancy-proxy-source" not found Nov 03 22:13:41.959451 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:13:41.959432 4298 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 03 22:13:41.959747 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:13:41.959486 4298 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b579ef-92ab-4fa5-a153-6be536d77795-tls-cert podName:08b579ef-92ab-4fa5-a153-6be536d77795 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:42.45947384 +0000 UTC m=+379.440207471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/08b579ef-92ab-4fa5-a153-6be536d77795-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" (UID: "08b579ef-92ab-4fa5-a153-6be536d77795") : secret "usage-logs-tenancy-proxy-tls" not found Nov 03 22:13:41.959747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.959535 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08b579ef-92ab-4fa5-a153-6be536d77795-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:41.960116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.960097 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08b579ef-92ab-4fa5-a153-6be536d77795-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:41.961342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:41.961318 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:42.017593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:42.017512 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9r9c\" (UniqueName: \"kubernetes.io/projected/08b579ef-92ab-4fa5-a153-6be536d77795-kube-api-access-r9r9c\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:42.250338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:42.250304 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:42.464384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:42.464344 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08b579ef-92ab-4fa5-a153-6be536d77795-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:42.464602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:42.464477 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/08b579ef-92ab-4fa5-a153-6be536d77795-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:42.465305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:42.465280 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/08b579ef-92ab-4fa5-a153-6be536d77795-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:42.466748 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:42.466731 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08b579ef-92ab-4fa5-a153-6be536d77795-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-v4vlc\" (UID: \"08b579ef-92ab-4fa5-a153-6be536d77795\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:42.663330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:42.663297 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:43.027336 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:13:43.027299 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b579ef_92ab_4fa5_a153_6be536d77795.slice/crio-bbaf47185b1e5cbfb75ca5a5455cf85b0df58c74f471d66b4095aab4e6aa4f9e WatchSource:0}: Error finding container bbaf47185b1e5cbfb75ca5a5455cf85b0df58c74f471d66b4095aab4e6aa4f9e: Status 404 returned error can't find the container with id bbaf47185b1e5cbfb75ca5a5455cf85b0df58c74f471d66b4095aab4e6aa4f9e Nov 03 22:13:43.029273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:43.029247 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc"] Nov 03 22:13:43.635626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:43.635591 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" event={"ID":"08b579ef-92ab-4fa5-a153-6be536d77795","Type":"ContainerStarted","Data":"bbaf47185b1e5cbfb75ca5a5455cf85b0df58c74f471d66b4095aab4e6aa4f9e"} Nov 03 22:13:47.561630 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:13:47.561544 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:13:56.702596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:56.702555 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" event={"ID":"08b579ef-92ab-4fa5-a153-6be536d77795","Type":"ContainerStarted","Data":"eac12d9686564a119c918c5ffc995ecc62afcf772a2a455d3c8fa38e33eb31a6"} Nov 03 22:13:56.703004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:56.702705 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:13:56.820121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:13:56.820026 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" podStartSLOduration=2.375947873 podStartE2EDuration="15.820011459s" totalImagesPullingTime="13.444063586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:56.819321494 +0000 UTC m=+393.800055147" watchObservedRunningTime="2026-11-03 22:13:56.820011459 +0000 UTC m=+393.800745188" Nov 03 22:14:04.603846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:04.603755 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:07.712988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:07.712949 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-v4vlc" Nov 03 22:14:12.682573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:12.682530 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:12.855503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:12.855459 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:13.031602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.031567 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:13.317992 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.317894 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:13.511067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.511040 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:13.579372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.579292 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ltvt5"] Nov 03 22:14:13.582785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.582768 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" Nov 03 22:14:13.657425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.657392 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fnhf4\"" Nov 03 22:14:13.686872 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.686840 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 22:14:13.688241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.688220 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ltvt5"] Nov 03 22:14:13.691166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.691149 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv"] Nov 03 22:14:13.694489 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.694470 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:14:13.705679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.705634 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:13.748752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.748724 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv"] Nov 03 22:14:13.749272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.749243 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-d2krk\"" Nov 03 22:14:13.749376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.749345 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 22:14:13.749935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.749918 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977-cert\") pod \"kserve-controller-manager-776cc66d47-ltvt5\" (UID: \"845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" Nov 03 22:14:13.749981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.749969 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xnn5\" (UniqueName: \"kubernetes.io/projected/845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977-kube-api-access-7xnn5\") pod \"kserve-controller-manager-776cc66d47-ltvt5\" (UID: \"845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" Nov 03 22:14:13.811635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.811606 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xhcp4"] Nov 03 22:14:13.814967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.814945 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" Nov 03 22:14:13.851095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.851029 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xnn5\" (UniqueName: \"kubernetes.io/projected/845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977-kube-api-access-7xnn5\") pod \"kserve-controller-manager-776cc66d47-ltvt5\" (UID: \"845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" Nov 03 22:14:13.851095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.851063 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d483dcc1-9a69-424e-b835-ac6c48087c1a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-csxkv\" (UID: \"d483dcc1-9a69-424e-b835-ac6c48087c1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:14:13.851095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.851081 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzvsp\" (UniqueName: \"kubernetes.io/projected/d483dcc1-9a69-424e-b835-ac6c48087c1a-kube-api-access-wzvsp\") pod \"llmisvc-controller-manager-869d5dcc5d-csxkv\" (UID: \"d483dcc1-9a69-424e-b835-ac6c48087c1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:14:13.851274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.851209 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977-cert\") pod \"kserve-controller-manager-776cc66d47-ltvt5\" (UID: \"845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" Nov 03 22:14:13.853745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.853717 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977-cert\") pod \"kserve-controller-manager-776cc66d47-ltvt5\" (UID: \"845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" Nov 03 22:14:13.868556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.868519 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-28422\"" Nov 03 22:14:13.868556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.868530 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 22:14:13.870168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.870147 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xhcp4"] Nov 03 22:14:13.909099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.909067 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xnn5\" (UniqueName: \"kubernetes.io/projected/845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977-kube-api-access-7xnn5\") pod \"kserve-controller-manager-776cc66d47-ltvt5\" (UID: \"845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" Nov 03 22:14:13.926704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.926677 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-jwnml"] Nov 03 22:14:13.930240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.930220 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:13.952710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.952677 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d483dcc1-9a69-424e-b835-ac6c48087c1a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-csxkv\" (UID: \"d483dcc1-9a69-424e-b835-ac6c48087c1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:14:13.952885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.952721 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzvsp\" (UniqueName: \"kubernetes.io/projected/d483dcc1-9a69-424e-b835-ac6c48087c1a-kube-api-access-wzvsp\") pod \"llmisvc-controller-manager-869d5dcc5d-csxkv\" (UID: \"d483dcc1-9a69-424e-b835-ac6c48087c1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:14:13.952885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.952766 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d9e9bcd-c138-4660-b91f-d76fc7c48e65-tls-certs\") pod \"model-serving-api-75bd89fdfc-xhcp4\" (UID: \"6d9e9bcd-c138-4660-b91f-d76fc7c48e65\") " pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" Nov 03 22:14:13.952885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.952794 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6pf\" (UniqueName: \"kubernetes.io/projected/6d9e9bcd-c138-4660-b91f-d76fc7c48e65-kube-api-access-gh6pf\") pod \"model-serving-api-75bd89fdfc-xhcp4\" (UID: \"6d9e9bcd-c138-4660-b91f-d76fc7c48e65\") " pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" Nov 03 22:14:13.955065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.955043 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d483dcc1-9a69-424e-b835-ac6c48087c1a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-csxkv\" (UID: \"d483dcc1-9a69-424e-b835-ac6c48087c1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:14:13.984309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:13.984279 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-jwnml"] Nov 03 22:14:14.010805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.010774 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzvsp\" (UniqueName: \"kubernetes.io/projected/d483dcc1-9a69-424e-b835-ac6c48087c1a-kube-api-access-wzvsp\") pod \"llmisvc-controller-manager-869d5dcc5d-csxkv\" (UID: \"d483dcc1-9a69-424e-b835-ac6c48087c1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:14:14.054023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.053990 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq9tk\" (UniqueName: \"kubernetes.io/projected/85f834ec-6594-48d1-87b8-2cc2299860e9-kube-api-access-rq9tk\") pod \"odh-model-controller-769875db56-jwnml\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:14.054187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.054060 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d9e9bcd-c138-4660-b91f-d76fc7c48e65-tls-certs\") pod \"model-serving-api-75bd89fdfc-xhcp4\" (UID: \"6d9e9bcd-c138-4660-b91f-d76fc7c48e65\") " pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" Nov 03 22:14:14.054187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.054113 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6pf\" (UniqueName: \"kubernetes.io/projected/6d9e9bcd-c138-4660-b91f-d76fc7c48e65-kube-api-access-gh6pf\") pod \"model-serving-api-75bd89fdfc-xhcp4\" (UID: \"6d9e9bcd-c138-4660-b91f-d76fc7c48e65\") " pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" Nov 03 22:14:14.054187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.054145 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-metrics-certs\") pod \"odh-model-controller-769875db56-jwnml\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:14.054187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.054187 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-cert\") pod \"odh-model-controller-769875db56-jwnml\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:14.056462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.056441 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d9e9bcd-c138-4660-b91f-d76fc7c48e65-tls-certs\") pod \"model-serving-api-75bd89fdfc-xhcp4\" (UID: \"6d9e9bcd-c138-4660-b91f-d76fc7c48e65\") " pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" Nov 03 22:14:14.069088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.069065 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 22:14:14.069195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.069116 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 22:14:14.069195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.069171 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-48bmq\"" Nov 03 22:14:14.117114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.117052 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6pf\" (UniqueName: \"kubernetes.io/projected/6d9e9bcd-c138-4660-b91f-d76fc7c48e65-kube-api-access-gh6pf\") pod \"model-serving-api-75bd89fdfc-xhcp4\" (UID: \"6d9e9bcd-c138-4660-b91f-d76fc7c48e65\") " pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" Nov 03 22:14:14.126798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.126774 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" Nov 03 22:14:14.154698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.154674 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-metrics-certs\") pod \"odh-model-controller-769875db56-jwnml\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:14.154842 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.154723 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-cert\") pod \"odh-model-controller-769875db56-jwnml\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:14.154842 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.154767 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq9tk\" (UniqueName: \"kubernetes.io/projected/85f834ec-6594-48d1-87b8-2cc2299860e9-kube-api-access-rq9tk\") pod \"odh-model-controller-769875db56-jwnml\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:14.157018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.156997 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-metrics-certs\") pod \"odh-model-controller-769875db56-jwnml\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:14.157201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.157180 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-cert\") pod \"odh-model-controller-769875db56-jwnml\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:14.195201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.195003 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" Nov 03 22:14:14.212605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.212573 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq9tk\" (UniqueName: \"kubernetes.io/projected/85f834ec-6594-48d1-87b8-2cc2299860e9-kube-api-access-rq9tk\") pod \"odh-model-controller-769875db56-jwnml\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:14.243174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.243144 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:14.306886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.306859 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:14:14.489995 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:14:14.489958 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9e9bcd_c138_4660_b91f_d76fc7c48e65.slice/crio-7a45ffcf323dfaf50dfa1b8819779703625c49b95c5002ee4410ca098a4c23b7 WatchSource:0}: Error finding container 7a45ffcf323dfaf50dfa1b8819779703625c49b95c5002ee4410ca098a4c23b7: Status 404 returned error can't find the container with id 7a45ffcf323dfaf50dfa1b8819779703625c49b95c5002ee4410ca098a4c23b7 Nov 03 22:14:14.491569 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.491541 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xhcp4"] Nov 03 22:14:14.584676 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:14:14.584622 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845f0e2e_e3d7_4ac7_ad6e_7c80b50f0977.slice/crio-00b7a722a60796f254c43b4e0f3e6ee9ceeb89a404d736050b724e410e305bfa WatchSource:0}: Error finding container 00b7a722a60796f254c43b4e0f3e6ee9ceeb89a404d736050b724e410e305bfa: Status 404 returned error can't find the container with id 00b7a722a60796f254c43b4e0f3e6ee9ceeb89a404d736050b724e410e305bfa Nov 03 22:14:14.587489 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.587459 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ltvt5"] Nov 03 22:14:14.618697 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:14:14.618662 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f834ec_6594_48d1_87b8_2cc2299860e9.slice/crio-01cbedd28cccbc93aa288fe4acbcba8ad78a42f477ab191fb9d0386ef48ad142 WatchSource:0}: Error finding container 01cbedd28cccbc93aa288fe4acbcba8ad78a42f477ab191fb9d0386ef48ad142: Status 404 returned error can't find the container with id 01cbedd28cccbc93aa288fe4acbcba8ad78a42f477ab191fb9d0386ef48ad142 Nov 03 22:14:14.620302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.620270 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-jwnml"] Nov 03 22:14:14.670748 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:14:14.670716 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd483dcc1_9a69_424e_b835_ac6c48087c1a.slice/crio-54c2cb702ec6cc81f0487aefc61ea10637a94339eef33dd7dc0cf95ca0f91c47 WatchSource:0}: Error finding container 54c2cb702ec6cc81f0487aefc61ea10637a94339eef33dd7dc0cf95ca0f91c47: Status 404 returned error can't find the container with id 54c2cb702ec6cc81f0487aefc61ea10637a94339eef33dd7dc0cf95ca0f91c47 Nov 03 22:14:14.673100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.673076 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv"] Nov 03 22:14:14.780892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.780806 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jwnml" event={"ID":"85f834ec-6594-48d1-87b8-2cc2299860e9","Type":"ContainerStarted","Data":"01cbedd28cccbc93aa288fe4acbcba8ad78a42f477ab191fb9d0386ef48ad142"} Nov 03 22:14:14.781955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.781930 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" event={"ID":"d483dcc1-9a69-424e-b835-ac6c48087c1a","Type":"ContainerStarted","Data":"54c2cb702ec6cc81f0487aefc61ea10637a94339eef33dd7dc0cf95ca0f91c47"} Nov 03 22:14:14.783032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.783008 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" event={"ID":"845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977","Type":"ContainerStarted","Data":"00b7a722a60796f254c43b4e0f3e6ee9ceeb89a404d736050b724e410e305bfa"} Nov 03 22:14:14.784051 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:14.784032 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" event={"ID":"6d9e9bcd-c138-4660-b91f-d76fc7c48e65","Type":"ContainerStarted","Data":"7a45ffcf323dfaf50dfa1b8819779703625c49b95c5002ee4410ca098a4c23b7"} Nov 03 22:14:15.004354 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:15.004305 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:15.297095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:15.297062 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:15.504505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:15.504472 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:15.690499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:15.690416 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:17.568669 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:14:17.568618 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:14:18.819870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:18.819494 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" event={"ID":"6d9e9bcd-c138-4660-b91f-d76fc7c48e65","Type":"ContainerStarted","Data":"f0e3b38285270cff0f94ca6c25ede79229c2894c0fa761e4c342d90174c354e1"} Nov 03 22:14:18.819870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:18.819586 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" Nov 03 22:14:18.961248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:18.960708 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" podStartSLOduration=2.213767702 podStartE2EDuration="5.960688661s" totalImagesPullingTime="3.746920959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:18.95838442 +0000 UTC m=+415.939118094" watchObservedRunningTime="2026-11-03 22:14:18.960688661 +0000 UTC m=+415.941422314" Nov 03 22:14:19.846667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:19.834182 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" event={"ID":"845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977","Type":"ContainerStarted","Data":"9b24d0cedd94de68d0f24f7881d5c30410a0948fa2cbcabf396c07fe8842b82b"} Nov 03 22:14:19.846667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:19.835056 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" Nov 03 22:14:19.846667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:19.837983 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jwnml" event={"ID":"85f834ec-6594-48d1-87b8-2cc2299860e9","Type":"ContainerStarted","Data":"6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2"} Nov 03 22:14:19.846667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:19.838027 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:19.951699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:19.951618 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" podStartSLOduration=1.921667319 podStartE2EDuration="6.951603298s" totalImagesPullingTime="5.029935979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:19.950847626 +0000 UTC m=+416.931581280" watchObservedRunningTime="2026-11-03 22:14:19.951603298 +0000 UTC m=+416.932337282" Nov 03 22:14:20.067300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:20.067250 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-jwnml" podStartSLOduration=2.030038288 podStartE2EDuration="7.067234619s" totalImagesPullingTime="5.037196331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:20.066628872 +0000 UTC m=+417.047362525" watchObservedRunningTime="2026-11-03 22:14:20.067234619 +0000 UTC m=+417.047968271" Nov 03 22:14:20.155026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:20.154928 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:20.337668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:20.333087 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:20.509500 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:20.509469 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:20.685512 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:20.685472 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:20.857740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:20.857658 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" event={"ID":"d483dcc1-9a69-424e-b835-ac6c48087c1a","Type":"ContainerStarted","Data":"1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0"} Nov 03 22:14:20.858138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:20.858083 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:14:20.909384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:20.909342 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:20.974518 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:20.974465 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" podStartSLOduration=2.783096779 podStartE2EDuration="7.974451526s" totalImagesPullingTime="5.191354747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:20.973284011 +0000 UTC m=+417.954017663" watchObservedRunningTime="2026-11-03 22:14:20.974451526 +0000 UTC m=+417.955185178" Nov 03 22:14:21.140748 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:21.138436 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:21.314805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:21.314770 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:21.491042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:21.491011 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:27.822590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:27.822558 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:28.098055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:28.097973 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:28.920146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:28.894088 4298 generic.go:356] "Generic (PLEG): container finished" podID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerID="ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb" exitCode=0 Nov 03 22:14:28.920146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:28.894163 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" event={"ID":"317f4b1e-43be-46a0-861e-f72df5c135ce","Type":"ContainerDied","Data":"ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb"} Nov 03 22:14:28.920146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:28.894532 4298 scope.go:122] "RemoveContainer" containerID="ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb" Nov 03 22:14:28.920146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:28.895566 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:14:28.920146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:28.895628 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:14:28.920146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:28.895682 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:14:29.852369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:29.852336 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-xhcp4" Nov 03 22:14:29.900604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:29.900571 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" event={"ID":"317f4b1e-43be-46a0-861e-f72df5c135ce","Type":"ContainerStarted","Data":"a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc"} Nov 03 22:14:30.845513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:30.845045 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:14:30.845513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:30.845180 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body= Nov 03 22:14:30.845513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:30.845242 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" Nov 03 22:14:30.860590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:30.860561 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:14:31.449702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.436515 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ml5tf"] Nov 03 22:14:31.450311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.449959 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" podUID="604d18f3-b84d-45e6-a840-61f673b79eb7" containerName="authorino" containerID="cri-o://56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692" gracePeriod=30 Nov 03 22:14:31.761206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.761180 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" Nov 03 22:14:31.844331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.843862 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body= Nov 03 22:14:31.844331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.843923 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" Nov 03 22:14:31.867097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.867066 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604d18f3-b84d-45e6-a840-61f673b79eb7-kube-api-access-jnzhl\" (UniqueName: \"kubernetes.io/projected/604d18f3-b84d-45e6-a840-61f673b79eb7-kube-api-access-jnzhl\") pod \"604d18f3-b84d-45e6-a840-61f673b79eb7\" (UID: \"604d18f3-b84d-45e6-a840-61f673b79eb7\") " Nov 03 22:14:31.869124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.869101 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604d18f3-b84d-45e6-a840-61f673b79eb7-kube-api-access-jnzhl" pod "604d18f3-b84d-45e6-a840-61f673b79eb7" (UID: "604d18f3-b84d-45e6-a840-61f673b79eb7"). InnerVolumeSpecName "kube-api-access-jnzhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:14:31.909995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.909966 4298 generic.go:356] "Generic (PLEG): container finished" podID="604d18f3-b84d-45e6-a840-61f673b79eb7" containerID="56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692" exitCode=0 Nov 03 22:14:31.910140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.910007 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" event={"ID":"604d18f3-b84d-45e6-a840-61f673b79eb7","Type":"ContainerDied","Data":"56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692"} Nov 03 22:14:31.910140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.910009 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" Nov 03 22:14:31.910140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.910044 4298 scope.go:122] "RemoveContainer" containerID="56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692" Nov 03 22:14:31.910140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.910034 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ml5tf" event={"ID":"604d18f3-b84d-45e6-a840-61f673b79eb7","Type":"ContainerDied","Data":"1f26d2901a96c21b39c70ae7f239d2e1f57aba7b7ece09c29660400ab08c2f1d"} Nov 03 22:14:31.918474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.918459 4298 scope.go:122] "RemoveContainer" containerID="56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692" Nov 03 22:14:31.918756 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:14:31.918728 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692\": container with ID starting with 56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692 not found: ID does not exist" containerID="56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692" Nov 03 22:14:31.918849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.918762 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692"} err="failed to get container status \"56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692\": rpc error: code = NotFound desc = could not find container \"56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692\": container with ID starting with 56237d6f30a11beed1039a1f579e8d72264b4d96a098380641cfb25401a71692 not found: ID does not exist" Nov 03 22:14:31.968438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:31.968358 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnzhl\" (UniqueName: \"kubernetes.io/projected/604d18f3-b84d-45e6-a840-61f673b79eb7-kube-api-access-jnzhl\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:32.075543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:32.075510 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ml5tf"] Nov 03 22:14:32.079413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:32.079392 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ml5tf"] Nov 03 22:14:32.844666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:32.844055 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body= Nov 03 22:14:32.844666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:32.844137 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" Nov 03 22:14:33.151787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:33.151696 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604d18f3-b84d-45e6-a840-61f673b79eb7" path="/var/lib/kubelet/pods/604d18f3-b84d-45e6-a840-61f673b79eb7/volumes" Nov 03 22:14:33.848669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:33.844024 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body= Nov 03 22:14:33.848669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:33.844089 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" Nov 03 22:14:34.843504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:34.843471 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body= Nov 03 22:14:34.844013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:34.843543 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" Nov 03 22:14:35.847047 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:35.847015 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:14:35.847423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:35.847292 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:14:35.848027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:35.848011 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:14:47.551661 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:14:47.551611 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:14:51.867138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:51.867107 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ltvt5" Nov 03 22:14:52.868263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:14:52.868233 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:15:00.171288 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.171249 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t"] Nov 03 22:15:00.171847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.171591 4298 cpu_manager.go:414] "containerMap: removing container" podUID="604d18f3-b84d-45e6-a840-61f673b79eb7" containerName="authorino" Nov 03 22:15:00.171847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.171602 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604d18f3-b84d-45e6-a840-61f673b79eb7" containerName="authorino" Nov 03 22:15:00.171847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.171689 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="604d18f3-b84d-45e6-a840-61f673b79eb7" containerName="authorino" Nov 03 22:15:00.178343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.178325 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" Nov 03 22:15:00.215439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.215409 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqvww\" (UniqueName: \"kubernetes.io/projected/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-kube-api-access-cqvww\") pod \"maas-api-key-cleanup-29895735-zl84t\" (UID: \"d92a7c4a-acf7-460c-ba7f-3a3eef795af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" Nov 03 22:15:00.215573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.215458 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895735-zl84t\" (UID: \"d92a7c4a-acf7-460c-ba7f-3a3eef795af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" Nov 03 22:15:00.226970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.226944 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t"] Nov 03 22:15:00.276556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.276525 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:15:00.316771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.316739 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqvww\" (UniqueName: \"kubernetes.io/projected/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-kube-api-access-cqvww\") pod \"maas-api-key-cleanup-29895735-zl84t\" (UID: \"d92a7c4a-acf7-460c-ba7f-3a3eef795af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" Nov 03 22:15:00.316950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.316778 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895735-zl84t\" (UID: \"d92a7c4a-acf7-460c-ba7f-3a3eef795af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" Nov 03 22:15:00.317439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.317414 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895735-zl84t\" (UID: \"d92a7c4a-acf7-460c-ba7f-3a3eef795af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" Nov 03 22:15:00.374543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.374505 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqvww\" (UniqueName: \"kubernetes.io/projected/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-kube-api-access-cqvww\") pod \"maas-api-key-cleanup-29895735-zl84t\" (UID: \"d92a7c4a-acf7-460c-ba7f-3a3eef795af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" Nov 03 22:15:00.490133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.490097 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" Nov 03 22:15:00.853759 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:15:00.853721 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92a7c4a_acf7_460c_ba7f_3a3eef795af0.slice/crio-7b00a0138e2b61be6fd9a50a34cb315bc026ae937e40466cb6f4b60e4cb99242 WatchSource:0}: Error finding container 7b00a0138e2b61be6fd9a50a34cb315bc026ae937e40466cb6f4b60e4cb99242: Status 404 returned error can't find the container with id 7b00a0138e2b61be6fd9a50a34cb315bc026ae937e40466cb6f4b60e4cb99242 Nov 03 22:15:00.855139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:00.855113 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t"] Nov 03 22:15:01.037773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:01.037743 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" event={"ID":"d92a7c4a-acf7-460c-ba7f-3a3eef795af0","Type":"ContainerStarted","Data":"7b00a0138e2b61be6fd9a50a34cb315bc026ae937e40466cb6f4b60e4cb99242"} Nov 03 22:15:04.936874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:04.936831 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:06.062325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:06.062284 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" event={"ID":"d92a7c4a-acf7-460c-ba7f-3a3eef795af0","Type":"ContainerStarted","Data":"dceb3ddaa2ccd52c152d4a912b25ecb12bc0106fb4d794fb016356d93e7270e9"} Nov 03 22:15:06.178043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:06.177985 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" podStartSLOduration=1.414632199 podStartE2EDuration="6.177971357s" totalImagesPullingTime="4.763339158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:06.177176501 +0000 UTC m=+463.157910156" watchObservedRunningTime="2026-11-03 22:15:06.177971357 +0000 UTC m=+463.158705007" Nov 03 22:15:07.068444 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:07.068408 4298 generic.go:356] "Generic (PLEG): container finished" podID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerID="dceb3ddaa2ccd52c152d4a912b25ecb12bc0106fb4d794fb016356d93e7270e9" exitCode=60 Nov 03 22:15:07.068975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:07.068461 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" event={"ID":"d92a7c4a-acf7-460c-ba7f-3a3eef795af0","Type":"ContainerDied","Data":"dceb3ddaa2ccd52c152d4a912b25ecb12bc0106fb4d794fb016356d93e7270e9"} Nov 03 22:15:07.068975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:07.068766 4298 scope.go:122] "RemoveContainer" containerID="dceb3ddaa2ccd52c152d4a912b25ecb12bc0106fb4d794fb016356d93e7270e9" Nov 03 22:15:08.075633 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:08.075546 4298 generic.go:356] "Generic (PLEG): container finished" podID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerID="67addfb6ac457eaf4031fedd4ef78e33597b59db20d5caa13579a84dc10b858f" exitCode=60 Nov 03 22:15:08.075633 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:08.075598 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" event={"ID":"d92a7c4a-acf7-460c-ba7f-3a3eef795af0","Type":"ContainerDied","Data":"67addfb6ac457eaf4031fedd4ef78e33597b59db20d5caa13579a84dc10b858f"} Nov 03 22:15:08.076132 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:08.075665 4298 scope.go:122] "RemoveContainer" containerID="dceb3ddaa2ccd52c152d4a912b25ecb12bc0106fb4d794fb016356d93e7270e9" Nov 03 22:15:08.076132 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:08.075865 4298 scope.go:122] "RemoveContainer" containerID="67addfb6ac457eaf4031fedd4ef78e33597b59db20d5caa13579a84dc10b858f" Nov 03 22:15:08.076132 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:15:08.076104 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895735-zl84t_odh-ai-gateway-infra(d92a7c4a-acf7-460c-ba7f-3a3eef795af0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" Nov 03 22:15:09.081633 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:09.081605 4298 scope.go:122] "RemoveContainer" containerID="67addfb6ac457eaf4031fedd4ef78e33597b59db20d5caa13579a84dc10b858f" Nov 03 22:15:09.082025 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:15:09.081837 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895735-zl84t_odh-ai-gateway-infra(d92a7c4a-acf7-460c-ba7f-3a3eef795af0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" Nov 03 22:15:17.552671 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:15:17.552601 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:15:19.127734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:19.127706 4298 scope.go:122] "RemoveContainer" containerID="67addfb6ac457eaf4031fedd4ef78e33597b59db20d5caa13579a84dc10b858f" Nov 03 22:15:20.129977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:20.129895 4298 generic.go:356] "Generic (PLEG): container finished" podID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerID="aaff5dc78a252b1fecd9159eb781c8059dea490e27cab0939c7873b4cc95dd12" exitCode=60 Nov 03 22:15:20.129977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:20.129964 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" event={"ID":"d92a7c4a-acf7-460c-ba7f-3a3eef795af0","Type":"ContainerDied","Data":"aaff5dc78a252b1fecd9159eb781c8059dea490e27cab0939c7873b4cc95dd12"} Nov 03 22:15:20.130397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:20.130013 4298 scope.go:122] "RemoveContainer" containerID="67addfb6ac457eaf4031fedd4ef78e33597b59db20d5caa13579a84dc10b858f" Nov 03 22:15:20.130397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:20.130254 4298 scope.go:122] "RemoveContainer" containerID="aaff5dc78a252b1fecd9159eb781c8059dea490e27cab0939c7873b4cc95dd12" Nov 03 22:15:20.130480 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:15:20.130421 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895735-zl84t_odh-ai-gateway-infra(d92a7c4a-acf7-460c-ba7f-3a3eef795af0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" Nov 03 22:15:20.135152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:20.135128 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:20.307091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:20.307052 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:21.256072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:21.256042 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t"] Nov 03 22:15:21.388892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:21.388868 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" Nov 03 22:15:21.508976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:21.508888 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-kube-api-access-cqvww\" (UniqueName: \"kubernetes.io/projected/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-kube-api-access-cqvww\") pod \"d92a7c4a-acf7-460c-ba7f-3a3eef795af0\" (UID: \"d92a7c4a-acf7-460c-ba7f-3a3eef795af0\") " Nov 03 22:15:21.508976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:21.508923 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-maas-api-service-ca\") pod \"d92a7c4a-acf7-460c-ba7f-3a3eef795af0\" (UID: \"d92a7c4a-acf7-460c-ba7f-3a3eef795af0\") " Nov 03 22:15:21.509383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:21.509353 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-maas-api-service-ca" pod "d92a7c4a-acf7-460c-ba7f-3a3eef795af0" (UID: "d92a7c4a-acf7-460c-ba7f-3a3eef795af0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:15:21.511088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:21.511055 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-kube-api-access-cqvww" pod "d92a7c4a-acf7-460c-ba7f-3a3eef795af0" (UID: "d92a7c4a-acf7-460c-ba7f-3a3eef795af0"). InnerVolumeSpecName "kube-api-access-cqvww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:21.609802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:21.609769 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqvww\" (UniqueName: \"kubernetes.io/projected/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-kube-api-access-cqvww\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:15:21.609802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:21.609798 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d92a7c4a-acf7-460c-ba7f-3a3eef795af0-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:15:22.140139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:22.140099 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" event={"ID":"d92a7c4a-acf7-460c-ba7f-3a3eef795af0","Type":"ContainerDied","Data":"7b00a0138e2b61be6fd9a50a34cb315bc026ae937e40466cb6f4b60e4cb99242"} Nov 03 22:15:22.140321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:22.140154 4298 scope.go:122] "RemoveContainer" containerID="aaff5dc78a252b1fecd9159eb781c8059dea490e27cab0939c7873b4cc95dd12" Nov 03 22:15:22.140321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:22.140116 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t" Nov 03 22:15:22.307594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:22.307559 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t"] Nov 03 22:15:22.312678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:22.312655 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-zl84t"] Nov 03 22:15:23.133990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:23.133960 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" path="/var/lib/kubelet/pods/d92a7c4a-acf7-460c-ba7f-3a3eef795af0/volumes" Nov 03 22:15:32.879307 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.879262 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l"] Nov 03 22:15:32.879866 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.879841 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerName="cleanup" Nov 03 22:15:32.879866 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.879858 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerName="cleanup" Nov 03 22:15:32.879985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.879876 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerName="cleanup" Nov 03 22:15:32.879985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.879884 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerName="cleanup" Nov 03 22:15:32.879985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.879907 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerName="cleanup" Nov 03 22:15:32.879985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.879914 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerName="cleanup" Nov 03 22:15:32.880172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.880016 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerName="cleanup" Nov 03 22:15:32.880172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.880028 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerName="cleanup" Nov 03 22:15:32.880172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.880040 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92a7c4a-acf7-460c-ba7f-3a3eef795af0" containerName="cleanup" Nov 03 22:15:32.884626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.884607 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" Nov 03 22:15:32.899895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:32.899867 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l"] Nov 03 22:15:33.014293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:33.014210 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2962c9c-7179-4c8b-8bc0-bae9e785b340-cert\") pod \"llmisvc-controller-manager-6466c554d8-hnz7l\" (UID: \"d2962c9c-7179-4c8b-8bc0-bae9e785b340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" Nov 03 22:15:33.014293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:33.014263 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hct\" (UniqueName: \"kubernetes.io/projected/d2962c9c-7179-4c8b-8bc0-bae9e785b340-kube-api-access-c7hct\") pod \"llmisvc-controller-manager-6466c554d8-hnz7l\" (UID: \"d2962c9c-7179-4c8b-8bc0-bae9e785b340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" Nov 03 22:15:33.115212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:33.115173 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2962c9c-7179-4c8b-8bc0-bae9e785b340-cert\") pod \"llmisvc-controller-manager-6466c554d8-hnz7l\" (UID: \"d2962c9c-7179-4c8b-8bc0-bae9e785b340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" Nov 03 22:15:33.115385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:33.115221 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hct\" (UniqueName: \"kubernetes.io/projected/d2962c9c-7179-4c8b-8bc0-bae9e785b340-kube-api-access-c7hct\") pod \"llmisvc-controller-manager-6466c554d8-hnz7l\" (UID: \"d2962c9c-7179-4c8b-8bc0-bae9e785b340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" Nov 03 22:15:33.117752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:33.117729 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2962c9c-7179-4c8b-8bc0-bae9e785b340-cert\") pod \"llmisvc-controller-manager-6466c554d8-hnz7l\" (UID: \"d2962c9c-7179-4c8b-8bc0-bae9e785b340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" Nov 03 22:15:33.173539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:33.173466 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hct\" (UniqueName: \"kubernetes.io/projected/d2962c9c-7179-4c8b-8bc0-bae9e785b340-kube-api-access-c7hct\") pod \"llmisvc-controller-manager-6466c554d8-hnz7l\" (UID: \"d2962c9c-7179-4c8b-8bc0-bae9e785b340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" Nov 03 22:15:33.197015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:33.196985 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" Nov 03 22:15:33.561760 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:15:33.561730 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2962c9c_7179_4c8b_8bc0_bae9e785b340.slice/crio-707d9ff46f51d4b781044cea94bda61b93f6675d25f00224bef8612de77b5b0f WatchSource:0}: Error finding container 707d9ff46f51d4b781044cea94bda61b93f6675d25f00224bef8612de77b5b0f: Status 404 returned error can't find the container with id 707d9ff46f51d4b781044cea94bda61b93f6675d25f00224bef8612de77b5b0f Nov 03 22:15:33.563239 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:33.563215 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l"] Nov 03 22:15:34.220092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:34.212874 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" event={"ID":"d2962c9c-7179-4c8b-8bc0-bae9e785b340","Type":"ContainerStarted","Data":"707d9ff46f51d4b781044cea94bda61b93f6675d25f00224bef8612de77b5b0f"} Nov 03 22:15:35.218438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:35.218396 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" event={"ID":"d2962c9c-7179-4c8b-8bc0-bae9e785b340","Type":"ContainerStarted","Data":"afb5ef978aa3777c1c888780cb0cc89fa48159982ec2df0fa6470a92a475da0e"} Nov 03 22:15:35.218611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:35.218494 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" Nov 03 22:15:35.334821 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:35.334766 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" podStartSLOduration=2.549790911 podStartE2EDuration="3.334751861s" totalImagesPullingTime="784.96095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:35.333287011 +0000 UTC m=+492.314020664" watchObservedRunningTime="2026-11-03 22:15:35.334751861 +0000 UTC m=+492.315485531" Nov 03 22:15:47.550909 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:15:47.550868 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:15:47.650966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:15:47.650934 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:05.291215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:05.291175 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:06.224937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:06.224909 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hnz7l" Nov 03 22:16:06.392615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:06.392584 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv"] Nov 03 22:16:06.393134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:06.392790 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" podUID="d483dcc1-9a69-424e-b835-ac6c48087c1a" containerName="manager" containerID="cri-o://1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0" gracePeriod=30 Nov 03 22:16:06.639312 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:06.639288 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:16:06.725869 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:06.725822 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d483dcc1-9a69-424e-b835-ac6c48087c1a-kube-api-access-wzvsp\" (UniqueName: \"kubernetes.io/projected/d483dcc1-9a69-424e-b835-ac6c48087c1a-kube-api-access-wzvsp\") pod \"d483dcc1-9a69-424e-b835-ac6c48087c1a\" (UID: \"d483dcc1-9a69-424e-b835-ac6c48087c1a\") " Nov 03 22:16:06.726055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:06.725902 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d483dcc1-9a69-424e-b835-ac6c48087c1a-cert\" (UniqueName: \"kubernetes.io/secret/d483dcc1-9a69-424e-b835-ac6c48087c1a-cert\") pod \"d483dcc1-9a69-424e-b835-ac6c48087c1a\" (UID: \"d483dcc1-9a69-424e-b835-ac6c48087c1a\") " Nov 03 22:16:06.728025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:06.727994 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d483dcc1-9a69-424e-b835-ac6c48087c1a-kube-api-access-wzvsp" pod "d483dcc1-9a69-424e-b835-ac6c48087c1a" (UID: "d483dcc1-9a69-424e-b835-ac6c48087c1a"). InnerVolumeSpecName "kube-api-access-wzvsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:06.728134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:06.728048 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d483dcc1-9a69-424e-b835-ac6c48087c1a-cert" pod "d483dcc1-9a69-424e-b835-ac6c48087c1a" (UID: "d483dcc1-9a69-424e-b835-ac6c48087c1a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:16:06.827193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:06.827113 4298 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d483dcc1-9a69-424e-b835-ac6c48087c1a-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:06.827193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:06.827142 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzvsp\" (UniqueName: \"kubernetes.io/projected/d483dcc1-9a69-424e-b835-ac6c48087c1a-kube-api-access-wzvsp\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:07.222215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:07.222176 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:07.357651 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:07.357616 4298 generic.go:356] "Generic (PLEG): container finished" podID="d483dcc1-9a69-424e-b835-ac6c48087c1a" containerID="1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0" exitCode=0 Nov 03 22:16:07.357825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:07.357685 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" Nov 03 22:16:07.357825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:07.357701 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" event={"ID":"d483dcc1-9a69-424e-b835-ac6c48087c1a","Type":"ContainerDied","Data":"1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0"} Nov 03 22:16:07.357825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:07.357748 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv" event={"ID":"d483dcc1-9a69-424e-b835-ac6c48087c1a","Type":"ContainerDied","Data":"54c2cb702ec6cc81f0487aefc61ea10637a94339eef33dd7dc0cf95ca0f91c47"} Nov 03 22:16:07.357825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:07.357769 4298 scope.go:122] "RemoveContainer" containerID="1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0" Nov 03 22:16:07.367082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:07.367061 4298 scope.go:122] "RemoveContainer" containerID="1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0" Nov 03 22:16:07.367350 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:16:07.367332 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0\": container with ID starting with 1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0 not found: ID does not exist" containerID="1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0" Nov 03 22:16:07.367393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:07.367361 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0"} err="failed to get container status \"1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0\": rpc error: code = NotFound desc = could not find container \"1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0\": container with ID starting with 1ad3fb0ffb413b025db38b1872f39c32368ab35051ddd9fbcc30ae9490fd14d0 not found: ID does not exist" Nov 03 22:16:07.416457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:07.416385 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:07.529285 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:07.529250 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv"] Nov 03 22:16:07.538963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:07.538928 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-csxkv"] Nov 03 22:16:09.133402 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:09.133371 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d483dcc1-9a69-424e-b835-ac6c48087c1a" path="/var/lib/kubelet/pods/d483dcc1-9a69-424e-b835-ac6c48087c1a/volumes" Nov 03 22:16:13.203753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:13.203706 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:17.552051 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:16:17.552016 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:16:24.233116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:24.233085 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:26.139483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:26.139452 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:30.371731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:30.371695 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:36.190661 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:36.190613 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:36.369343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:36.369308 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:36.984557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:36.984516 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:42.473014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:42.472981 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:45.729383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:45.729340 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:47.551862 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:16:47.551827 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:16:49.824555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:49.824520 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:53.231373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:53.231331 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:56.642610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:16:56.642576 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:05.637493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:05.637401 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:11.375710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:11.375661 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:17.553018 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:17:17.552973 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:17:24.431324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.431284 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd"] Nov 03 22:17:24.431897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.431877 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d483dcc1-9a69-424e-b835-ac6c48087c1a" containerName="manager" Nov 03 22:17:24.431976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.431898 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d483dcc1-9a69-424e-b835-ac6c48087c1a" containerName="manager" Nov 03 22:17:24.432031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.432008 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d483dcc1-9a69-424e-b835-ac6c48087c1a" containerName="manager" Nov 03 22:17:24.435149 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.435128 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.527621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.527588 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.527827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.527629 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk2t6\" (UniqueName: \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-kube-api-access-vk2t6\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.527827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.527765 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.527827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.527793 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95448530-22d3-444e-abde-6b6b873f139f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.527949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.527829 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.527986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.527938 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95448530-22d3-444e-abde-6b6b873f139f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.528035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.528013 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.528112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.528099 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.528147 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.528133 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.541718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.541692 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd"] Nov 03 22:17:24.629127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.629076 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk2t6\" (UniqueName: \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-kube-api-access-vk2t6\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.629299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.629152 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.629365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.629329 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95448530-22d3-444e-abde-6b6b873f139f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.629422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.629384 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.629485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.629443 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95448530-22d3-444e-abde-6b6b873f139f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.629485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.629473 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.629606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.629573 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.629606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.629586 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.629861 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.629837 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.630001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.629981 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.630283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.630260 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.630683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.630533 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.630683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.630633 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95448530-22d3-444e-abde-6b6b873f139f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.630980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.630940 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.632514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.632460 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.632954 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.632931 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95448530-22d3-444e-abde-6b6b873f139f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.686966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.686900 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk2t6\" (UniqueName: \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-kube-api-access-vk2t6\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.688088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.688066 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4szjd\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:24.748215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:24.748185 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:25.163039 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:17:25.163011 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95448530_22d3_444e_abde_6b6b873f139f.slice/crio-49b615ba27ac043a8130bb31b8861b4c34cf449e6752d39eba38003066f1f23a WatchSource:0}: Error finding container 49b615ba27ac043a8130bb31b8861b4c34cf449e6752d39eba38003066f1f23a: Status 404 returned error can't find the container with id 49b615ba27ac043a8130bb31b8861b4c34cf449e6752d39eba38003066f1f23a Nov 03 22:17:25.165795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:25.164226 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd"] Nov 03 22:17:25.168005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:25.167975 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:17:25.168084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:25.168037 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:17:25.168084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:25.168067 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:17:25.749778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:25.749730 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" event={"ID":"95448530-22d3-444e-abde-6b6b873f139f","Type":"ContainerStarted","Data":"f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83"} Nov 03 22:17:25.750193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:25.749796 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" event={"ID":"95448530-22d3-444e-abde-6b6b873f139f","Type":"ContainerStarted","Data":"49b615ba27ac043a8130bb31b8861b4c34cf449e6752d39eba38003066f1f23a"} Nov 03 22:17:25.868435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:25.868364 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" podStartSLOduration=1.868348525 podStartE2EDuration="1.868348525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:25.867842087 +0000 UTC m=+602.848575737" watchObservedRunningTime="2026-11-03 22:17:25.868348525 +0000 UTC m=+602.849082178" Nov 03 22:17:26.748824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:26.748786 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:26.750867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:26.750835 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4szjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 22:17:26.751221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:26.750885 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" podUID="95448530-22d3-444e-abde-6b6b873f139f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 22:17:27.749124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:27.749013 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4szjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 22:17:27.749124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:27.749073 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" podUID="95448530-22d3-444e-abde-6b6b873f139f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 22:17:28.055872 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:28.055797 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:28.264619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:28.264587 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:28.749049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:28.749013 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4szjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 22:17:28.749258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:28.749070 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" podUID="95448530-22d3-444e-abde-6b6b873f139f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 22:17:29.749571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:29.749530 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4szjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 22:17:29.750013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:29.749589 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" podUID="95448530-22d3-444e-abde-6b6b873f139f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 22:17:30.749118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:30.749073 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4szjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 22:17:30.749316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:30.749148 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" podUID="95448530-22d3-444e-abde-6b6b873f139f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 22:17:31.749351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:31.749317 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4szjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 22:17:31.749755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:31.749373 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" podUID="95448530-22d3-444e-abde-6b6b873f139f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 22:17:32.749203 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:32.749168 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4szjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 22:17:32.749379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:32.749220 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" podUID="95448530-22d3-444e-abde-6b6b873f139f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 22:17:33.752379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:33.752348 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:33.752893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:33.752734 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:33.753306 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:33.753284 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:17:38.071557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.071506 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:38.252883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.252842 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:38.588363 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.588330 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-7b8kb"] Nov 03 22:17:38.592401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.592374 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" Nov 03 22:17:38.643803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.643769 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-7b8kb"] Nov 03 22:17:38.657039 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.657007 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416fda64-c90d-4b05-b7e2-4f963d168b50-plugins-config-volume\") pod \"payload-processing-7bb9489676-7b8kb\" (UID: \"416fda64-c90d-4b05-b7e2-4f963d168b50\") " pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" Nov 03 22:17:38.657217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.657073 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchb7\" (UniqueName: \"kubernetes.io/projected/416fda64-c90d-4b05-b7e2-4f963d168b50-kube-api-access-gchb7\") pod \"payload-processing-7bb9489676-7b8kb\" (UID: \"416fda64-c90d-4b05-b7e2-4f963d168b50\") " pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" Nov 03 22:17:38.693129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.693094 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sdlvs\"" Nov 03 22:17:38.696635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.696610 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 22:17:38.758566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.758525 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416fda64-c90d-4b05-b7e2-4f963d168b50-plugins-config-volume\") pod \"payload-processing-7bb9489676-7b8kb\" (UID: \"416fda64-c90d-4b05-b7e2-4f963d168b50\") " pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" Nov 03 22:17:38.758768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.758587 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchb7\" (UniqueName: \"kubernetes.io/projected/416fda64-c90d-4b05-b7e2-4f963d168b50-kube-api-access-gchb7\") pod \"payload-processing-7bb9489676-7b8kb\" (UID: \"416fda64-c90d-4b05-b7e2-4f963d168b50\") " pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" Nov 03 22:17:38.759336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.759312 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416fda64-c90d-4b05-b7e2-4f963d168b50-plugins-config-volume\") pod \"payload-processing-7bb9489676-7b8kb\" (UID: \"416fda64-c90d-4b05-b7e2-4f963d168b50\") " pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" Nov 03 22:17:38.816656 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.816603 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchb7\" (UniqueName: \"kubernetes.io/projected/416fda64-c90d-4b05-b7e2-4f963d168b50-kube-api-access-gchb7\") pod \"payload-processing-7bb9489676-7b8kb\" (UID: \"416fda64-c90d-4b05-b7e2-4f963d168b50\") " pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" Nov 03 22:17:38.908384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:38.908293 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" Nov 03 22:17:39.274888 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:17:39.274847 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416fda64_c90d_4b05_b7e2_4f963d168b50.slice/crio-0f5555f2f15b5ca9ef83ff18060ef28ffbd83888896506547e18a178c59e9598 WatchSource:0}: Error finding container 0f5555f2f15b5ca9ef83ff18060ef28ffbd83888896506547e18a178c59e9598: Status 404 returned error can't find the container with id 0f5555f2f15b5ca9ef83ff18060ef28ffbd83888896506547e18a178c59e9598 Nov 03 22:17:39.276237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:39.276213 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-7b8kb"] Nov 03 22:17:39.277409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:39.277389 4298 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:17:39.809855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:39.809820 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" event={"ID":"416fda64-c90d-4b05-b7e2-4f963d168b50","Type":"ContainerStarted","Data":"0f5555f2f15b5ca9ef83ff18060ef28ffbd83888896506547e18a178c59e9598"} Nov 03 22:17:42.826226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:42.826193 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" event={"ID":"416fda64-c90d-4b05-b7e2-4f963d168b50","Type":"ContainerStarted","Data":"bf0d6593b5f915813b3af2d3daa0cf6e5ed376f004cb830f036cc375f7e13b19"} Nov 03 22:17:42.826624 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:42.826272 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" Nov 03 22:17:42.941945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:42.941890 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" podStartSLOduration=2.141005719 podStartE2EDuration="4.941875095s" totalImagesPullingTime="2.800869376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:42.941002613 +0000 UTC m=+619.921736267" watchObservedRunningTime="2026-11-03 22:17:42.941875095 +0000 UTC m=+619.922608748" Nov 03 22:17:47.553068 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:17:47.553027 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:17:53.833802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:17:53.833770 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-7b8kb" Nov 03 22:18:01.968689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:01.968628 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:02.415125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:02.415018 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:03.941318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:03.941283 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:04.209707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:04.209673 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:05.517166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:05.517108 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:05.698335 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:05.698295 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:05.983123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:05.983088 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:12.015984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:12.015940 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:12.184517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:12.184477 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:17.551306 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:18:17.551261 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:18:30.286567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:30.286528 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:30.463133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:30.463088 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:30.880362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:30.880303 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:31.083944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.083909 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj"] Nov 03 22:18:31.095549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.095514 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.132447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.132347 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.132447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.132409 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.133984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.132552 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.133984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.132618 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.133984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.132674 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.133984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.132700 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.133984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.132718 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.133984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.132755 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxgbt\" (UniqueName: \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-kube-api-access-pxgbt\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.133984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.132796 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.144443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.144413 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj"] Nov 03 22:18:31.147712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.147624 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:31.193725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.193690 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f81f8f-openshift-default-dockercfg-mvg86\"" Nov 03 22:18:31.234235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.234201 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.234427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.234261 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.234427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.234305 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.234427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.234334 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.234427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.234349 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.234427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.234379 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.234427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.234408 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.234768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.234440 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxgbt\" (UniqueName: \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-kube-api-access-pxgbt\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.234768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.234489 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.234768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.234740 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.234924 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.234875 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.235448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.235118 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.235448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.235154 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.235448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.235124 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.238304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.238278 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.239126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.238884 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.292210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.292136 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.292620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.292231 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxgbt\" (UniqueName: \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-kube-api-access-pxgbt\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.327727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.327690 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:31.409602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.409516 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:31.550389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.550344 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:31.827229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.827192 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:31.830600 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:18:31.830569 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cfe907_f90c_4555_86a4_5eee9c7dcc00.slice/crio-d48d54f5868b5f5b2576fd82f568ab5ddb61f1e73f3fdd1b0648aa0be09af0a6 WatchSource:0}: Error finding container d48d54f5868b5f5b2576fd82f568ab5ddb61f1e73f3fdd1b0648aa0be09af0a6: Status 404 returned error can't find the container with id d48d54f5868b5f5b2576fd82f568ab5ddb61f1e73f3fdd1b0648aa0be09af0a6 Nov 03 22:18:31.832388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.832368 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj"] Nov 03 22:18:31.833037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.833010 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:18:31.833106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.833069 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:18:31.833106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:31.833097 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:18:32.030850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.030815 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv"] Nov 03 22:18:32.034730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.034706 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.048780 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.048749 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" event={"ID":"b4cfe907-f90c-4555-86a4-5eee9c7dcc00","Type":"ContainerStarted","Data":"2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec"} Nov 03 22:18:32.048913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.048802 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" event={"ID":"b4cfe907-f90c-4555-86a4-5eee9c7dcc00","Type":"ContainerStarted","Data":"d48d54f5868b5f5b2576fd82f568ab5ddb61f1e73f3fdd1b0648aa0be09af0a6"} Nov 03 22:18:32.089281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.089199 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-906673-openshift-default-dockercfg-cwtcb\"" Nov 03 22:18:32.091134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.091088 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv"] Nov 03 22:18:32.105089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.105054 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:32.146303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.146274 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.146477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.146313 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.146477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.146332 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.146477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.146352 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.146619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.146515 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.146619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.146541 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4hsb\" (UniqueName: \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-kube-api-access-q4hsb\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.146619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.146594 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.146771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.146738 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.146807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.146775 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.249625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.248300 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.249625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.248358 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hsb\" (UniqueName: \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-kube-api-access-q4hsb\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.249625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.248420 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.249625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.248455 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.249625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.248486 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.249625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.248558 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.249625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.248596 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.249625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.248616 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.249625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.248636 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.249625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.249219 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.250405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.250378 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.250531 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.250477 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.250596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.250520 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.251017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.250664 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.251676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.251636 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.253150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.253130 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.282682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.282601 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" podStartSLOduration=1.282583931 podStartE2EDuration="1.282583931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:32.280884278 +0000 UTC m=+669.261617932" watchObservedRunningTime="2026-11-03 22:18:32.282583931 +0000 UTC m=+669.263317585" Nov 03 22:18:32.296878 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.296849 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:32.306323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.306295 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.306582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.306558 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hsb\" (UniqueName: \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-kube-api-access-q4hsb\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.349252 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.349156 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:32.412735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.411379 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:32.421562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.421535 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:32.469689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.469638 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:32.655544 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.655462 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:32.671264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.671225 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2"] Nov 03 22:18:32.676417 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.676395 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.746078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.746042 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2"] Nov 03 22:18:32.746247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.746197 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f7e6db-openshift-default-dockercfg-dfr7z\"" Nov 03 22:18:32.752957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.752929 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.753106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.752981 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.753160 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.753126 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.753216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.753199 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.753257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.753233 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwnn\" (UniqueName: \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-kube-api-access-6bwnn\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.753301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.753266 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.753451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.753429 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.753503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.753490 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.753614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.753585 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.798357 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:18:32.798328 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2811ac9d_28b8_4f35_b0b0_82079bd5d3b2.slice/crio-22f654dc8f8c15744b2f6af01c2318623ce1d53fa709b728a61a54ca7c437a35 WatchSource:0}: Error finding container 22f654dc8f8c15744b2f6af01c2318623ce1d53fa709b728a61a54ca7c437a35: Status 404 returned error can't find the container with id 22f654dc8f8c15744b2f6af01c2318623ce1d53fa709b728a61a54ca7c437a35 Nov 03 22:18:32.799447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.799422 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv"] Nov 03 22:18:32.800952 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.800926 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:18:32.801037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.800988 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:18:32.801037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.801020 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:18:32.854154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.854128 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.854257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.854178 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.854257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.854234 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.854373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.854262 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.854373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.854291 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwnn\" (UniqueName: \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-kube-api-access-6bwnn\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.854373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.854318 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.854373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.854347 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.854570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.854388 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.854570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.854441 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.854723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.854575 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.855282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.854802 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.855282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.855139 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.855282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.855227 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.855282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.855257 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.857138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.857082 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.857303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.857285 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.897736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.897678 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:32.913254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.913158 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.913558 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.913537 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwnn\" (UniqueName: \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-kube-api-access-6bwnn\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:32.989658 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:32.989546 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:33.090812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.073479 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:33.090812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.077657 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" event={"ID":"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2","Type":"ContainerStarted","Data":"3a27aaef636276f008c93be040595d260221e3fe8ebb0ce1e70b9d80f15a712d"} Nov 03 22:18:33.090812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.077712 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:33.090812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.077740 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" event={"ID":"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2","Type":"ContainerStarted","Data":"22f654dc8f8c15744b2f6af01c2318623ce1d53fa709b728a61a54ca7c437a35"} Nov 03 22:18:33.090812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.079276 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:18:33.198481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.198433 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" podStartSLOduration=2.198418598 podStartE2EDuration="2.198418598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:33.198214226 +0000 UTC m=+670.178947880" watchObservedRunningTime="2026-11-03 22:18:33.198418598 +0000 UTC m=+670.179152252" Nov 03 22:18:33.253945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.253909 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:33.349409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.349377 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:33.355762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.355735 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:33.418582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.418557 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2"] Nov 03 22:18:33.419939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.419906 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:18:33.420032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.419982 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:18:33.420032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.420012 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:18:33.440375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:33.440351 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:34.082079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:34.082040 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" event={"ID":"d6cba416-b4fa-41a0-9a0f-9e3249d52c92","Type":"ContainerStarted","Data":"f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433"} Nov 03 22:18:34.082256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:34.082104 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" event={"ID":"d6cba416-b4fa-41a0-9a0f-9e3249d52c92","Type":"ContainerStarted","Data":"0a01f26dd91dbb2674ac8a26050ed15fac51687ff95bf6f2e96d4f0038798a54"} Nov 03 22:18:34.082315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:34.082282 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:34.082997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:34.082976 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:18:34.200307 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:34.200259 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" podStartSLOduration=2.200244918 podStartE2EDuration="2.200244918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:34.199623789 +0000 UTC m=+671.180357453" watchObservedRunningTime="2026-11-03 22:18:34.200244918 +0000 UTC m=+671.180978571" Nov 03 22:18:34.989812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:34.989778 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:34.994933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:34.994907 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:35.086993 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:35.086958 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:35.087559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:35.087537 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:18:35.651841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:35.651806 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:35.838254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:35.838220 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:36.178418 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:36.178379 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:36.367637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:36.367594 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:36.551589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:36.551560 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:36.936010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:36.935934 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:37.244792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:37.244763 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:37.668602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:37.668500 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:37.855577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:37.855533 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:39.326386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:39.326352 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:39.515579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:39.515548 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:39.887523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:39.887492 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:40.063840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:40.063808 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:40.922718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:40.922685 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:41.129043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:41.129011 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:41.310922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:41.310890 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:41.500551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:41.500502 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:41.684532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:41.684440 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:42.534048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:42.534015 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:42.738327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:42.738288 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:45.021850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:45.021817 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:45.200919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:45.200883 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:46.472591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:46.472558 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:46.650896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:46.650863 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:46.826016 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:46.825918 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:47.552943 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:18:47.552906 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:18:47.617201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:47.617167 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:47.795112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:47.795075 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:47.985108 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:47.985072 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9"] Nov 03 22:18:47.988351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:47.988333 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:18:48.045448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:48.045419 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-906673\"" Nov 03 22:18:48.096505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:48.096478 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-906673-dockercfg-lcq2m\"" Nov 03 22:18:48.099571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:48.099541 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9"] Nov 03 22:18:48.113611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:48.113579 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf7n6\" (UniqueName: \"kubernetes.io/projected/ecea227c-1b19-4a6c-b115-77d3987557f8-kube-api-access-tf7n6\") pod \"payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9\" (UID: \"ecea227c-1b19-4a6c-b115-77d3987557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:18:48.113812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:48.113703 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecea227c-1b19-4a6c-b115-77d3987557f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9\" (UID: \"ecea227c-1b19-4a6c-b115-77d3987557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:18:48.214479 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:48.214445 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf7n6\" (UniqueName: \"kubernetes.io/projected/ecea227c-1b19-4a6c-b115-77d3987557f8-kube-api-access-tf7n6\") pod \"payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9\" (UID: \"ecea227c-1b19-4a6c-b115-77d3987557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:18:48.214847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:48.214500 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecea227c-1b19-4a6c-b115-77d3987557f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9\" (UID: \"ecea227c-1b19-4a6c-b115-77d3987557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:18:48.215469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:48.215445 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecea227c-1b19-4a6c-b115-77d3987557f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9\" (UID: \"ecea227c-1b19-4a6c-b115-77d3987557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:18:48.275607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:48.275529 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf7n6\" (UniqueName: \"kubernetes.io/projected/ecea227c-1b19-4a6c-b115-77d3987557f8-kube-api-access-tf7n6\") pod \"payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9\" (UID: \"ecea227c-1b19-4a6c-b115-77d3987557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:18:48.301292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:48.301258 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:18:48.666379 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:18:48.666334 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecea227c_1b19_4a6c_b115_77d3987557f8.slice/crio-a634c7408fda99f11b3c3c4f79b9a596c2119c2167051aeb24e4b9b5c8eefb44 WatchSource:0}: Error finding container a634c7408fda99f11b3c3c4f79b9a596c2119c2167051aeb24e4b9b5c8eefb44: Status 404 returned error can't find the container with id a634c7408fda99f11b3c3c4f79b9a596c2119c2167051aeb24e4b9b5c8eefb44 Nov 03 22:18:48.667898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:48.667871 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9"] Nov 03 22:18:49.158345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:49.158311 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" event={"ID":"ecea227c-1b19-4a6c-b115-77d3987557f8","Type":"ContainerStarted","Data":"0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0"} Nov 03 22:18:49.158538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:49.158359 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" event={"ID":"ecea227c-1b19-4a6c-b115-77d3987557f8","Type":"ContainerStarted","Data":"a634c7408fda99f11b3c3c4f79b9a596c2119c2167051aeb24e4b9b5c8eefb44"} Nov 03 22:18:49.158538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:49.158448 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:18:49.275056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:49.275004 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" podStartSLOduration=2.274987864 podStartE2EDuration="2.274987864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:49.273350232 +0000 UTC m=+686.254083885" watchObservedRunningTime="2026-11-03 22:18:49.274987864 +0000 UTC m=+686.255721517" Nov 03 22:18:50.222491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.222448 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:50.403071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.403037 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:50.756316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.756272 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5"] Nov 03 22:18:50.760001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.759977 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:18:50.814005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.813973 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5"] Nov 03 22:18:50.842403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.842370 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a08a87-7277-4fe5-817d-90a33f64e8cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5\" (UID: \"62a08a87-7277-4fe5-817d-90a33f64e8cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:18:50.842554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.842507 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mztwv\" (UniqueName: \"kubernetes.io/projected/62a08a87-7277-4fe5-817d-90a33f64e8cb-kube-api-access-mztwv\") pod \"payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5\" (UID: \"62a08a87-7277-4fe5-817d-90a33f64e8cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:18:50.861890 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.861863 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-6a6b7f\"" Nov 03 22:18:50.863718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.863699 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-6a6b7f-dockercfg-dcsr2\"" Nov 03 22:18:50.944120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.944082 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a08a87-7277-4fe5-817d-90a33f64e8cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5\" (UID: \"62a08a87-7277-4fe5-817d-90a33f64e8cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:18:50.944290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.944180 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mztwv\" (UniqueName: \"kubernetes.io/projected/62a08a87-7277-4fe5-817d-90a33f64e8cb-kube-api-access-mztwv\") pod \"payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5\" (UID: \"62a08a87-7277-4fe5-817d-90a33f64e8cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:18:50.944884 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:50.944865 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a08a87-7277-4fe5-817d-90a33f64e8cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5\" (UID: \"62a08a87-7277-4fe5-817d-90a33f64e8cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:18:51.002119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:51.002083 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mztwv\" (UniqueName: \"kubernetes.io/projected/62a08a87-7277-4fe5-817d-90a33f64e8cb-kube-api-access-mztwv\") pod \"payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5\" (UID: \"62a08a87-7277-4fe5-817d-90a33f64e8cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:18:51.072359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:51.072265 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:18:51.437980 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:18:51.437944 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a08a87_7277_4fe5_817d_90a33f64e8cb.slice/crio-334b928e9b51d8093ac64111a2fc008d46a319fbc642eee47c34a733465cfb77 WatchSource:0}: Error finding container 334b928e9b51d8093ac64111a2fc008d46a319fbc642eee47c34a733465cfb77: Status 404 returned error can't find the container with id 334b928e9b51d8093ac64111a2fc008d46a319fbc642eee47c34a733465cfb77 Nov 03 22:18:51.441692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:51.441668 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5"] Nov 03 22:18:51.536170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:51.536137 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:51.712638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:51.712608 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:51.928767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:51.928736 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:52.173271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:52.173195 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" event={"ID":"62a08a87-7277-4fe5-817d-90a33f64e8cb","Type":"ContainerStarted","Data":"9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00"} Nov 03 22:18:52.173271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:52.173241 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" event={"ID":"62a08a87-7277-4fe5-817d-90a33f64e8cb","Type":"ContainerStarted","Data":"334b928e9b51d8093ac64111a2fc008d46a319fbc642eee47c34a733465cfb77"} Nov 03 22:18:52.173448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:52.173408 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:18:52.290491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:52.290438 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" podStartSLOduration=2.290422483 podStartE2EDuration="2.290422483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:52.288284273 +0000 UTC m=+689.269017927" watchObservedRunningTime="2026-11-03 22:18:52.290422483 +0000 UTC m=+689.271156136" Nov 03 22:18:52.960754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:52.960717 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:53.141448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:53.141402 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:53.356077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:53.355983 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:53.530036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:53.530002 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:55.234794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.234762 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:55.476615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.476578 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:55.614630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.614549 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8"] Nov 03 22:18:55.617933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.617917 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:18:55.671509 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.671481 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-41bb2b\"" Nov 03 22:18:55.725337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.725289 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8"] Nov 03 22:18:55.783289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.783224 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c"] Nov 03 22:18:55.788657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.788622 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:18:55.794042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.794017 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2sm\" (UniqueName: \"kubernetes.io/projected/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-kube-api-access-rk2sm\") pod \"payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8\" (UID: \"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:18:55.794165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.794054 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8\" (UID: \"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:18:55.824064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.824040 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c"] Nov 03 22:18:55.843652 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.843611 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-41bb2b-dockercfg-k4c7k\"" Nov 03 22:18:55.895162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.895080 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2sm\" (UniqueName: \"kubernetes.io/projected/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-kube-api-access-rk2sm\") pod \"payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8\" (UID: \"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:18:55.895162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.895120 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3317aadb-8f90-46e2-be55-fa9b81238900-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c\" (UID: \"3317aadb-8f90-46e2-be55-fa9b81238900\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:18:55.895162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.895140 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br2qh\" (UniqueName: \"kubernetes.io/projected/3317aadb-8f90-46e2-be55-fa9b81238900-kube-api-access-br2qh\") pod \"payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c\" (UID: \"3317aadb-8f90-46e2-be55-fa9b81238900\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:18:55.895162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.895160 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8\" (UID: \"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:18:55.896090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.896061 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8\" (UID: \"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:18:55.955104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.955071 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2sm\" (UniqueName: \"kubernetes.io/projected/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-kube-api-access-rk2sm\") pod \"payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8\" (UID: \"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:18:55.996800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.996765 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3317aadb-8f90-46e2-be55-fa9b81238900-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c\" (UID: \"3317aadb-8f90-46e2-be55-fa9b81238900\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:18:55.996957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.996804 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br2qh\" (UniqueName: \"kubernetes.io/projected/3317aadb-8f90-46e2-be55-fa9b81238900-kube-api-access-br2qh\") pod \"payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c\" (UID: \"3317aadb-8f90-46e2-be55-fa9b81238900\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:18:55.997472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:55.997453 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3317aadb-8f90-46e2-be55-fa9b81238900-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c\" (UID: \"3317aadb-8f90-46e2-be55-fa9b81238900\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:18:56.054813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:56.054781 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2qh\" (UniqueName: \"kubernetes.io/projected/3317aadb-8f90-46e2-be55-fa9b81238900-kube-api-access-br2qh\") pod \"payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c\" (UID: \"3317aadb-8f90-46e2-be55-fa9b81238900\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:18:56.102080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:56.102044 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:18:56.232031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:56.231996 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:18:56.320015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:56.319971 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:56.469118 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:18:56.469092 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3317aadb_8f90_46e2_be55_fa9b81238900.slice/crio-006d6bcbc880e8b7a06cbccd74000aec22dd63ead037a7209fe681b0600123cd WatchSource:0}: Error finding container 006d6bcbc880e8b7a06cbccd74000aec22dd63ead037a7209fe681b0600123cd: Status 404 returned error can't find the container with id 006d6bcbc880e8b7a06cbccd74000aec22dd63ead037a7209fe681b0600123cd Nov 03 22:18:56.471401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:56.471378 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c"] Nov 03 22:18:56.597683 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:18:56.597633 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aedc98d_0cc8_492e_ad70_6b7b5a661b1f.slice/crio-aa6df2b5ae630da80b7f12afb53f988e19749af9a1f799f7c2ab6be0048b4324 WatchSource:0}: Error finding container aa6df2b5ae630da80b7f12afb53f988e19749af9a1f799f7c2ab6be0048b4324: Status 404 returned error can't find the container with id aa6df2b5ae630da80b7f12afb53f988e19749af9a1f799f7c2ab6be0048b4324 Nov 03 22:18:56.599836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:56.599812 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8"] Nov 03 22:18:57.197614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:57.197577 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" event={"ID":"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f","Type":"ContainerStarted","Data":"375dd0b04c1997bce72541d9498111c3996816b8280f6cb8d10cd680513add7c"} Nov 03 22:18:57.197837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:57.197634 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" event={"ID":"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f","Type":"ContainerStarted","Data":"aa6df2b5ae630da80b7f12afb53f988e19749af9a1f799f7c2ab6be0048b4324"} Nov 03 22:18:57.197837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:57.197682 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:18:57.199168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:57.199143 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" event={"ID":"3317aadb-8f90-46e2-be55-fa9b81238900","Type":"ContainerStarted","Data":"a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0"} Nov 03 22:18:57.199264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:57.199191 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" event={"ID":"3317aadb-8f90-46e2-be55-fa9b81238900","Type":"ContainerStarted","Data":"006d6bcbc880e8b7a06cbccd74000aec22dd63ead037a7209fe681b0600123cd"} Nov 03 22:18:57.199264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:57.199221 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:18:57.312700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:57.312629 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" podStartSLOduration=2.312616097 podStartE2EDuration="2.312616097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:57.312173347 +0000 UTC m=+694.292907000" watchObservedRunningTime="2026-11-03 22:18:57.312616097 +0000 UTC m=+694.293349750" Nov 03 22:18:57.431783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:57.431713 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" podStartSLOduration=2.431692731 podStartE2EDuration="2.431692731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:57.430213218 +0000 UTC m=+694.410946929" watchObservedRunningTime="2026-11-03 22:18:57.431692731 +0000 UTC m=+694.412426398" Nov 03 22:18:57.872836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:57.872805 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:58.050910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.050869 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:58.254191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.254156 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2"] Nov 03 22:18:58.258014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.257989 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:18:58.311545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.311515 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2"] Nov 03 22:18:58.360277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.360232 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f61714\"" Nov 03 22:18:58.360466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.360421 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f61714-dockercfg-2k27n\"" Nov 03 22:18:58.419652 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.419612 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93671667-ad10-4586-8b80-1bb0f18b3a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2\" (UID: \"93671667-ad10-4586-8b80-1bb0f18b3a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:18:58.419847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.419823 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v987z\" (UniqueName: \"kubernetes.io/projected/93671667-ad10-4586-8b80-1bb0f18b3a2f-kube-api-access-v987z\") pod \"payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2\" (UID: \"93671667-ad10-4586-8b80-1bb0f18b3a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:18:58.520709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.520596 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93671667-ad10-4586-8b80-1bb0f18b3a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2\" (UID: \"93671667-ad10-4586-8b80-1bb0f18b3a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:18:58.521089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.520733 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v987z\" (UniqueName: \"kubernetes.io/projected/93671667-ad10-4586-8b80-1bb0f18b3a2f-kube-api-access-v987z\") pod \"payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2\" (UID: \"93671667-ad10-4586-8b80-1bb0f18b3a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:18:58.521353 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.521333 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93671667-ad10-4586-8b80-1bb0f18b3a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2\" (UID: \"93671667-ad10-4586-8b80-1bb0f18b3a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:18:58.580366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.580329 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v987z\" (UniqueName: \"kubernetes.io/projected/93671667-ad10-4586-8b80-1bb0f18b3a2f-kube-api-access-v987z\") pod \"payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2\" (UID: \"93671667-ad10-4586-8b80-1bb0f18b3a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:18:58.871246 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:58.871160 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:18:59.235189 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:18:59.235160 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93671667_ad10_4586_8b80_1bb0f18b3a2f.slice/crio-93faf86f5ad50c594682d624ef3280170ff9f62b97719547171f3e502019c8f0 WatchSource:0}: Error finding container 93faf86f5ad50c594682d624ef3280170ff9f62b97719547171f3e502019c8f0: Status 404 returned error can't find the container with id 93faf86f5ad50c594682d624ef3280170ff9f62b97719547171f3e502019c8f0 Nov 03 22:18:59.237502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:18:59.237476 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2"] Nov 03 22:19:00.166152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:00.166122 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:19:00.216384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:00.216351 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" event={"ID":"93671667-ad10-4586-8b80-1bb0f18b3a2f","Type":"ContainerStarted","Data":"175c253b7035b434cd67a76dbd64b37ac54b9b626403670f42a8fcbaa68ba7dd"} Nov 03 22:19:00.216571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:00.216412 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" event={"ID":"93671667-ad10-4586-8b80-1bb0f18b3a2f","Type":"ContainerStarted","Data":"93faf86f5ad50c594682d624ef3280170ff9f62b97719547171f3e502019c8f0"} Nov 03 22:19:00.216571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:00.216439 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:19:00.417791 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:00.417677 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" podStartSLOduration=2.4176624589999998 podStartE2EDuration="2.417662459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:00.416782874 +0000 UTC m=+697.397516527" watchObservedRunningTime="2026-11-03 22:19:00.417662459 +0000 UTC m=+697.398396109" Nov 03 22:19:00.861527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:00.861496 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:01.052824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:01.052758 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:01.249619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:01.249586 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:02.985734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:02.985704 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:03.172657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.172623 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:03.180399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.180374 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:19:03.677601 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.677562 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d"] Nov 03 22:19:03.681788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.681766 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:19:03.757292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.757258 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d"] Nov 03 22:19:03.781131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.781092 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z7r9\" (UniqueName: \"kubernetes.io/projected/1022b4bf-b218-4df7-a0f4-951e5711896f-kube-api-access-8z7r9\") pod \"payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d\" (UID: \"1022b4bf-b218-4df7-a0f4-951e5711896f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:19:03.781131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.781133 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1022b4bf-b218-4df7-a0f4-951e5711896f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d\" (UID: \"1022b4bf-b218-4df7-a0f4-951e5711896f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:19:03.805216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.805180 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f7e6db-dockercfg-htx7h\"" Nov 03 22:19:03.805390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.805236 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f7e6db\"" Nov 03 22:19:03.881949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.881912 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z7r9\" (UniqueName: \"kubernetes.io/projected/1022b4bf-b218-4df7-a0f4-951e5711896f-kube-api-access-8z7r9\") pod \"payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d\" (UID: \"1022b4bf-b218-4df7-a0f4-951e5711896f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:19:03.881949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.881956 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1022b4bf-b218-4df7-a0f4-951e5711896f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d\" (UID: \"1022b4bf-b218-4df7-a0f4-951e5711896f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:19:03.882771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.882751 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1022b4bf-b218-4df7-a0f4-951e5711896f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d\" (UID: \"1022b4bf-b218-4df7-a0f4-951e5711896f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:19:03.949225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.949194 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z7r9\" (UniqueName: \"kubernetes.io/projected/1022b4bf-b218-4df7-a0f4-951e5711896f-kube-api-access-8z7r9\") pod \"payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d\" (UID: \"1022b4bf-b218-4df7-a0f4-951e5711896f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:19:03.995498 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:03.995454 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:19:04.382939 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:19:04.382912 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1022b4bf_b218_4df7_a0f4_951e5711896f.slice/crio-ccf6d1acb2f846475eb74057ad59f05e0bc4ec33ec98994e7496bcbfc69b02f5 WatchSource:0}: Error finding container ccf6d1acb2f846475eb74057ad59f05e0bc4ec33ec98994e7496bcbfc69b02f5: Status 404 returned error can't find the container with id ccf6d1acb2f846475eb74057ad59f05e0bc4ec33ec98994e7496bcbfc69b02f5 Nov 03 22:19:04.383824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:04.383790 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d"] Nov 03 22:19:05.274659 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:05.274615 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" event={"ID":"1022b4bf-b218-4df7-a0f4-951e5711896f","Type":"ContainerStarted","Data":"d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa"} Nov 03 22:19:05.275174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:05.274688 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" event={"ID":"1022b4bf-b218-4df7-a0f4-951e5711896f","Type":"ContainerStarted","Data":"ccf6d1acb2f846475eb74057ad59f05e0bc4ec33ec98994e7496bcbfc69b02f5"} Nov 03 22:19:05.275174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:05.274813 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:19:05.405391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:05.405337 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" podStartSLOduration=2.405322861 podStartE2EDuration="2.405322861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:05.402313136 +0000 UTC m=+702.383046788" watchObservedRunningTime="2026-11-03 22:19:05.405322861 +0000 UTC m=+702.386056513" Nov 03 22:19:05.864796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:05.864762 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:06.380263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:06.380229 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:06.560868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:06.560786 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:08.207934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:08.207905 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:19:08.208406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:08.207982 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:19:10.195182 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:10.195139 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:10.301623 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:10.301594 4298 generic.go:356] "Generic (PLEG): container finished" podID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerID="3a27aaef636276f008c93be040595d260221e3fe8ebb0ce1e70b9d80f15a712d" exitCode=0 Nov 03 22:19:10.301812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:10.301672 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" event={"ID":"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2","Type":"ContainerDied","Data":"3a27aaef636276f008c93be040595d260221e3fe8ebb0ce1e70b9d80f15a712d"} Nov 03 22:19:10.302069 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:10.302054 4298 scope.go:122] "RemoveContainer" containerID="3a27aaef636276f008c93be040595d260221e3fe8ebb0ce1e70b9d80f15a712d" Nov 03 22:19:10.303485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:10.303461 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:19:10.303552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:10.303523 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:19:10.303587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:10.303556 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:19:10.409876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:10.409847 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:10.589188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:10.589109 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:10.774162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:10.768164 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:11.274578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:11.274538 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:19:11.308514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:11.308481 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" event={"ID":"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2","Type":"ContainerStarted","Data":"24318ed91e9a2ac3b4f2fa5d1d2efd3736deeb49f338e55bd65ab31c4be98052"} Nov 03 22:19:11.352309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:11.352274 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:19:11.352704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:11.352679 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 22:19:11.352909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:11.352869 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 22:19:12.353670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:12.353044 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 22:19:12.353670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:12.353105 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 22:19:13.357666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:13.351253 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 22:19:13.357666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:13.351309 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 22:19:14.357958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:14.357906 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 22:19:14.358586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:14.358556 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 22:19:15.268695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:15.268651 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:15.349618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:15.349581 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 22:19:15.349866 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:15.349679 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 22:19:15.456664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:15.456523 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:15.652729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:15.652366 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:16.208257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:16.208222 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:16.284553 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:16.284521 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:19:16.350271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:16.350229 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 22:19:16.350469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:16.350297 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 22:19:16.416763 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:16.416725 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:17.353897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:17.353863 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:19:17.354414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:17.354395 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:19:17.355093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:17.355070 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:19:17.556556 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:19:17.556521 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:19:18.078389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:18.078364 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 22:19:18.078556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:18.078414 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 22:19:18.370528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:18.370423 4298 generic.go:356] "Generic (PLEG): container finished" podID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerID="2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec" exitCode=0 Nov 03 22:19:18.370528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:18.370495 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" event={"ID":"b4cfe907-f90c-4555-86a4-5eee9c7dcc00","Type":"ContainerDied","Data":"2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec"} Nov 03 22:19:18.371076 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:18.371058 4298 scope.go:122] "RemoveContainer" containerID="2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec" Nov 03 22:19:18.372421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:18.372399 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:19:18.372484 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:18.372461 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:19:18.372523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:18.372491 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:19:19.381189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:19.381157 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" event={"ID":"b4cfe907-f90c-4555-86a4-5eee9c7dcc00","Type":"ContainerStarted","Data":"b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b"} Nov 03 22:19:19.409795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:19.409763 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:19:19.409983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:19.409958 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 22:19:19.410048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:19.410006 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 22:19:20.098828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:20.098796 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:20.409946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:20.409869 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 22:19:20.409946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:20.409919 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 22:19:20.551426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:20.551308 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:21.410886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:21.410850 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 22:19:21.411293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:21.410903 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 22:19:22.409877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:22.409843 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 22:19:22.410049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:22.409891 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 22:19:23.410030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:23.409991 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 22:19:23.410426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:23.410049 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 22:19:24.410658 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:24.410598 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 22:19:24.411070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:24.410697 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 22:19:24.954552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:24.954521 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:25.133735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:25.133681 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:25.305847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:25.305736 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:25.414204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:25.414170 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:19:25.414639 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:25.414471 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:19:25.415081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:25.415063 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:19:26.505470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:26.505429 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:27.233770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:27.233738 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:27.693884 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:27.693793 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:30.128571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:30.128538 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:35.476721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:35.476685 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:37.173522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:37.173467 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:37.750994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:37.750958 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:37.941244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:37.941212 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:40.511120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:40.511074 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:41.044167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:41.044134 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:41.498208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:41.498175 4298 generic.go:356] "Generic (PLEG): container finished" podID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerID="f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433" exitCode=0 Nov 03 22:19:41.498414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:41.498244 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" event={"ID":"d6cba416-b4fa-41a0-9a0f-9e3249d52c92","Type":"ContainerDied","Data":"f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433"} Nov 03 22:19:41.498662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:41.498625 4298 scope.go:122] "RemoveContainer" containerID="f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433" Nov 03 22:19:41.499946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:41.499919 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:19:41.500021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:41.499987 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:19:41.500058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:41.500018 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:19:42.504723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:42.504688 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" event={"ID":"d6cba416-b4fa-41a0-9a0f-9e3249d52c92","Type":"ContainerStarted","Data":"d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829"} Nov 03 22:19:42.990168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:42.990128 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:19:42.990392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:42.990374 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 22:19:42.990444 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:42.990415 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 22:19:43.651594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.651560 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt"] Nov 03 22:19:43.655476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.655452 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.708764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.708731 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt"] Nov 03 22:19:43.745200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.734267 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-tmp-dir\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.746507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.746465 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-home\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.748913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.748887 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s48hv\" (UniqueName: \"kubernetes.io/projected/d48a9429-f937-41c1-b434-d7fbef05ae0c-kube-api-access-s48hv\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.749093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.749077 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48a9429-f937-41c1-b434-d7fbef05ae0c-tls-certs\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.749234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.749219 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-model-cache\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.749371 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.749355 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-dshm\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.756921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.756891 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"default-dockercfg-rpq9b\"" Nov 03 22:19:43.756921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.756910 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"kube-root-ca.crt\"" Nov 03 22:19:43.757070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.756917 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"openshift-service-ca.crt\"" Nov 03 22:19:43.759777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.759756 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"simulator-w2-906673-kserve-self-signed-certs\"" Nov 03 22:19:43.778729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.778700 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:43.859307 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.859271 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-model-cache\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.859613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.859580 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-dshm\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.859830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.859808 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-tmp-dir\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.860121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.860098 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-home\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.860274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.860256 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s48hv\" (UniqueName: \"kubernetes.io/projected/d48a9429-f937-41c1-b434-d7fbef05ae0c-kube-api-access-s48hv\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.860424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.860407 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48a9429-f937-41c1-b434-d7fbef05ae0c-tls-certs\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.861960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.861939 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-tmp-dir\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.862179 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.862031 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-home\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.862323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.862125 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-model-cache\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.871820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.871785 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-dshm\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.872217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.872191 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48a9429-f937-41c1-b434-d7fbef05ae0c-tls-certs\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.919926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.919854 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s48hv\" (UniqueName: \"kubernetes.io/projected/d48a9429-f937-41c1-b434-d7fbef05ae0c-kube-api-access-s48hv\") pod \"simulator-w2-906673-kserve-675c5c9bfb-hlwdt\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.967210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.967155 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:43.990144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.990105 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 22:19:43.990305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:43.990171 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 22:19:44.375549 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:19:44.375513 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd48a9429_f937_41c1_b434_d7fbef05ae0c.slice/crio-c1e965d5af967dd3b6de24399963677f2a02bf885475c2f5538d288878cfce80 WatchSource:0}: Error finding container c1e965d5af967dd3b6de24399963677f2a02bf885475c2f5538d288878cfce80: Status 404 returned error can't find the container with id c1e965d5af967dd3b6de24399963677f2a02bf885475c2f5538d288878cfce80 Nov 03 22:19:44.378049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:44.378025 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt"] Nov 03 22:19:44.515766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:44.515730 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" event={"ID":"d48a9429-f937-41c1-b434-d7fbef05ae0c","Type":"ContainerStarted","Data":"c1e965d5af967dd3b6de24399963677f2a02bf885475c2f5538d288878cfce80"} Nov 03 22:19:44.990269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:44.990238 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 22:19:44.990694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:44.990301 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 22:19:45.658783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:45.654672 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:45.990213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:45.990167 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 22:19:45.990413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:45.990236 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 22:19:46.351443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:46.351413 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:46.557514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:46.557427 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" event={"ID":"d48a9429-f937-41c1-b434-d7fbef05ae0c","Type":"ContainerStarted","Data":"9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c"} Nov 03 22:19:46.676067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:46.676009 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podStartSLOduration=1.776880156 podStartE2EDuration="3.675994458s" totalImagesPullingTime="1.899114302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:46.674322035 +0000 UTC m=+743.655055682" watchObservedRunningTime="2026-11-03 22:19:46.675994458 +0000 UTC m=+743.656728111" Nov 03 22:19:46.913237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:46.913155 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:46.968113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:46.968080 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:46.990555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:46.990517 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 22:19:46.990974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:46.990573 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 22:19:47.103600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:47.103573 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:47.577522 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:19:47.577479 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:19:48.146328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:48.146295 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:19:48.146854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:48.146677 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:19:48.147839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:48.147812 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:19:50.466371 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:50.466342 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:52.688002 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.687965 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx"] Nov 03 22:19:52.691896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.691876 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.747352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.747314 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tls-certs\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.747543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.747394 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-dshm\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.747543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.747431 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tmp-dir\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.747664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.747543 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-home\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.747664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.747569 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4fm\" (UniqueName: \"kubernetes.io/projected/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-kube-api-access-zp4fm\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.747784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.747711 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-model-cache\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.747872 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.747850 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx"] Nov 03 22:19:52.796582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.796553 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"default-dockercfg-kdmdn\"" Nov 03 22:19:52.796773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.796592 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"simulator-w3-41bb2b-kserve-self-signed-certs\"" Nov 03 22:19:52.813927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.813897 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:52.847790 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.847757 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"openshift-service-ca.crt\"" Nov 03 22:19:52.847790 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.847780 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"kube-root-ca.crt\"" Nov 03 22:19:52.848184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.848168 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-model-cache\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.848224 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.848201 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tls-certs\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.848258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.848235 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-dshm\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.848258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.848250 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tmp-dir\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.848354 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.848303 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-home\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.848354 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.848332 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4fm\" (UniqueName: \"kubernetes.io/projected/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-kube-api-access-zp4fm\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.848701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.848678 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-model-cache\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.848790 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.848698 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-home\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.848790 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.848718 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tmp-dir\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.850678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.850632 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-dshm\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.850951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.850931 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tls-certs\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:52.906404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:52.906366 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4fm\" (UniqueName: \"kubernetes.io/projected/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-kube-api-access-zp4fm\") pod \"simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:53.003813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:53.003771 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:53.373629 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:19:53.373598 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod293038f2_6fc8_4c35_a1fb_d9ac2b02553d.slice/crio-24d86deedce6bff8fa60d451149bf192305f9361955b08918508115b4300f5c8 WatchSource:0}: Error finding container 24d86deedce6bff8fa60d451149bf192305f9361955b08918508115b4300f5c8: Status 404 returned error can't find the container with id 24d86deedce6bff8fa60d451149bf192305f9361955b08918508115b4300f5c8 Nov 03 22:19:53.375535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:53.375499 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx"] Nov 03 22:19:53.593373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:53.593337 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" event={"ID":"293038f2-6fc8-4c35-a1fb-d9ac2b02553d","Type":"ContainerStarted","Data":"31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023"} Nov 03 22:19:53.593373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:53.593378 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" event={"ID":"293038f2-6fc8-4c35-a1fb-d9ac2b02553d","Type":"ContainerStarted","Data":"24d86deedce6bff8fa60d451149bf192305f9361955b08918508115b4300f5c8"} Nov 03 22:19:53.710963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:53.710914 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podStartSLOduration=1.710898544 podStartE2EDuration="1.710898544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:53.708748927 +0000 UTC m=+750.689482579" watchObservedRunningTime="2026-11-03 22:19:53.710898544 +0000 UTC m=+750.691632226" Nov 03 22:19:53.967831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:53.967748 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:53.981174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:53.981147 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:54.004483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:54.004439 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:19:54.610049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:54.610019 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:19:55.071927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:55.071893 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:58.690137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:58.690098 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:59.239314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:59.239276 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:59.429275 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:59.429247 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:59.871331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:19:59.871303 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:02.990001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:02.989920 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:03.004937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:03.004890 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:20:03.018780 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:03.018751 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:20:03.538273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:03.538242 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:03.654736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:03.654703 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:20:03.721117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:03.721079 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:04.781528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:04.781491 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:06.743843 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:06.743807 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:09.286696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:09.286660 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:11.361867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:11.361831 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:13.704587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:13.704556 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:14.094816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:14.094719 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:14.305488 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:14.305459 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:14.486694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:14.486635 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:16.288455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.288420 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp"] Nov 03 22:20:16.298039 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.298004 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.350727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.350693 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp"] Nov 03 22:20:16.352748 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.352722 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"kube-root-ca.crt\"" Nov 03 22:20:16.398493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.398456 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"openshift-service-ca.crt\"" Nov 03 22:20:16.398661 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.398538 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"simulator-w5-f81f8f-kserve-self-signed-certs\"" Nov 03 22:20:16.398661 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.398629 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"default-dockercfg-hcclc\"" Nov 03 22:20:16.404935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.404914 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-tmp-dir\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.405034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.404959 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5w9\" (UniqueName: \"kubernetes.io/projected/026f1bf5-559b-492e-a9f4-a433644d790d-kube-api-access-lf5w9\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.405034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.404981 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-model-cache\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.405113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.405097 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-home\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.405149 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.405130 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026f1bf5-559b-492e-a9f4-a433644d790d-tls-certs\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.405193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.405176 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-dshm\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.417255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.417206 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:16.506357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.506325 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026f1bf5-559b-492e-a9f4-a433644d790d-tls-certs\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.506357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.506361 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-dshm\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.506604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.506443 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-tmp-dir\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.506604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.506482 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5w9\" (UniqueName: \"kubernetes.io/projected/026f1bf5-559b-492e-a9f4-a433644d790d-kube-api-access-lf5w9\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.506604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.506499 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-model-cache\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.506604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.506515 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-home\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.506974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.506948 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-tmp-dir\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.507073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.507042 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-home\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.507231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.507211 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-model-cache\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.509051 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.509028 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026f1bf5-559b-492e-a9f4-a433644d790d-tls-certs\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.509051 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.509041 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-dshm\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.566958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.566868 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5w9\" (UniqueName: \"kubernetes.io/projected/026f1bf5-559b-492e-a9f4-a433644d790d-kube-api-access-lf5w9\") pod \"simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.614340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.614299 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:16.991112 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:20:16.991085 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod026f1bf5_559b_492e_a9f4_a433644d790d.slice/crio-62c9323ef45fa45aa71280c91b924568b8327219a051288e6f4cbaf372d91b5e WatchSource:0}: Error finding container 62c9323ef45fa45aa71280c91b924568b8327219a051288e6f4cbaf372d91b5e: Status 404 returned error can't find the container with id 62c9323ef45fa45aa71280c91b924568b8327219a051288e6f4cbaf372d91b5e Nov 03 22:20:16.992145 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:16.992120 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp"] Nov 03 22:20:17.554611 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:20:17.554572 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:20:17.720197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:17.720165 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" event={"ID":"026f1bf5-559b-492e-a9f4-a433644d790d","Type":"ContainerStarted","Data":"a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2"} Nov 03 22:20:17.720462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:17.720205 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" event={"ID":"026f1bf5-559b-492e-a9f4-a433644d790d","Type":"ContainerStarted","Data":"62c9323ef45fa45aa71280c91b924568b8327219a051288e6f4cbaf372d91b5e"} Nov 03 22:20:17.838288 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:17.838170 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podStartSLOduration=1.8381553099999999 podStartE2EDuration="1.83815531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:17.836184522 +0000 UTC m=+774.816918176" watchObservedRunningTime="2026-11-03 22:20:17.83815531 +0000 UTC m=+774.818888962" Nov 03 22:20:18.615039 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:18.614965 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:18.689705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:18.689677 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:23.187478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:23.187441 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:26.614945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.614906 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:26.627867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.627838 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:26.734437 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.734405 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s"] Nov 03 22:20:26.740390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.740366 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:26.781579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.781550 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:20:26.787125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.787076 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s"] Nov 03 22:20:26.794106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.794078 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"openshift-service-ca.crt\"" Nov 03 22:20:26.794106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.794078 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"kube-root-ca.crt\"" Nov 03 22:20:26.794106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.794112 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"default-dockercfg-fr8gq\"" Nov 03 22:20:26.794106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.794152 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"simulator-w6-6a6b7f-kserve-self-signed-certs\"" Nov 03 22:20:26.918209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.918106 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-model-cache\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:26.918209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.918166 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dldn9\" (UniqueName: \"kubernetes.io/projected/fee74c97-aedc-4db7-a744-c3a7af678d05-kube-api-access-dldn9\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:26.918412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.918266 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee74c97-aedc-4db7-a744-c3a7af678d05-tls-certs\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:26.918460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.918408 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-home\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:26.918516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.918497 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-tmp-dir\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:26.918555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:26.918529 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-dshm\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.019473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.019440 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-home\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.019687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.019492 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-tmp-dir\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.019687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.019514 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-dshm\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.019687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.019575 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-model-cache\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.019687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.019620 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dldn9\" (UniqueName: \"kubernetes.io/projected/fee74c97-aedc-4db7-a744-c3a7af678d05-kube-api-access-dldn9\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.019687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.019676 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee74c97-aedc-4db7-a744-c3a7af678d05-tls-certs\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.019979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.019917 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-home\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.020020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.019977 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-tmp-dir\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.020567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.020540 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-model-cache\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.022002 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.021974 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-dshm\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.022131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.022115 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee74c97-aedc-4db7-a744-c3a7af678d05-tls-certs\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.077784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.077750 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dldn9\" (UniqueName: \"kubernetes.io/projected/fee74c97-aedc-4db7-a744-c3a7af678d05-kube-api-access-dldn9\") pod \"simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.123347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.123314 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:27.352926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.352871 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:27.702740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.702706 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:27.727477 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:20:27.727451 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfee74c97_aedc_4db7_a744_c3a7af678d05.slice/crio-9f2a5347da424d9814369d4114a2357e1888257b60acbd27934e4f691975d19d WatchSource:0}: Error finding container 9f2a5347da424d9814369d4114a2357e1888257b60acbd27934e4f691975d19d: Status 404 returned error can't find the container with id 9f2a5347da424d9814369d4114a2357e1888257b60acbd27934e4f691975d19d Nov 03 22:20:27.729967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.729948 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s"] Nov 03 22:20:27.774974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:27.774935 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" event={"ID":"fee74c97-aedc-4db7-a744-c3a7af678d05","Type":"ContainerStarted","Data":"9f2a5347da424d9814369d4114a2357e1888257b60acbd27934e4f691975d19d"} Nov 03 22:20:28.781322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:28.781280 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" event={"ID":"fee74c97-aedc-4db7-a744-c3a7af678d05","Type":"ContainerStarted","Data":"1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d"} Nov 03 22:20:28.897230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:28.897171 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podStartSLOduration=2.897159439 podStartE2EDuration="2.897159439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:28.896423311 +0000 UTC m=+785.877156964" watchObservedRunningTime="2026-11-03 22:20:28.897159439 +0000 UTC m=+785.877893093" Nov 03 22:20:29.353443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:29.353408 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:32.189307 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:32.189274 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:36.790495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:36.790418 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:37.353814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:37.353744 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:37.366501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:37.366467 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:37.838531 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:37.838502 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:20:40.192391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:40.192353 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:41.279959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:41.279931 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:45.781235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:45.781206 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:47.552094 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:20:47.552057 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:20:50.376630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:50.376590 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:53.849105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:53.849073 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:54.915563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:54.915532 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:59.379800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:20:59.379765 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:02.048527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.048483 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv"] Nov 03 22:21:02.053601 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.053562 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.106932 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.106893 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv"] Nov 03 22:21:02.155002 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.154974 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"premium-w2-906673-kserve-self-signed-certs\"" Nov 03 22:21:02.184909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.184876 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:02.188142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.188107 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tmp-dir\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.188238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.188158 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-model-cache\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.188310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.188283 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tls-certs\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.188410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.188392 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-dshm\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.188476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.188455 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-home\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.188519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.188504 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rktz\" (UniqueName: \"kubernetes.io/projected/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-kube-api-access-6rktz\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.290103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.290066 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-dshm\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.290313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.290123 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-home\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.290313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.290164 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rktz\" (UniqueName: \"kubernetes.io/projected/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-kube-api-access-6rktz\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.290313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.290185 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tmp-dir\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.290313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.290213 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-model-cache\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.290313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.290281 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tls-certs\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.290694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.290631 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-home\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.290800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.290702 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tmp-dir\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.290853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.290818 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-model-cache\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.292919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.292887 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tls-certs\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.292919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.292909 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-dshm\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.361909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.361836 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rktz\" (UniqueName: \"kubernetes.io/projected/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-kube-api-access-6rktz\") pod \"premium-w2-906673-kserve-f4d448dbf-9xnfv\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.367759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.367725 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:02.740423 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:21:02.740394 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cbb21e0_adc8_4a3f_ae50_ebd48828d4d2.slice/crio-042d9e550b8b72e1ac20b7bc1b038c6441e5c0cf13657ac84a06194000330db0 WatchSource:0}: Error finding container 042d9e550b8b72e1ac20b7bc1b038c6441e5c0cf13657ac84a06194000330db0: Status 404 returned error can't find the container with id 042d9e550b8b72e1ac20b7bc1b038c6441e5c0cf13657ac84a06194000330db0 Nov 03 22:21:02.742839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.742814 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv"] Nov 03 22:21:02.956936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.956889 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" event={"ID":"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2","Type":"ContainerStarted","Data":"7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8"} Nov 03 22:21:02.956936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:02.956933 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" event={"ID":"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2","Type":"ContainerStarted","Data":"042d9e550b8b72e1ac20b7bc1b038c6441e5c0cf13657ac84a06194000330db0"} Nov 03 22:21:03.074899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:03.074773 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podStartSLOduration=1.074760279 podStartE2EDuration="1.074760279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:03.072827061 +0000 UTC m=+820.053560711" watchObservedRunningTime="2026-11-03 22:21:03.074760279 +0000 UTC m=+820.055493931" Nov 03 22:21:03.368482 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:03.368387 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:03.993578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:03.993548 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:07.095346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:07.095307 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:07.281394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:07.281365 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:08.481840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:08.481806 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:12.006454 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:12.006419 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:12.368584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:12.368493 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:12.381085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:12.381058 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:12.985297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:12.985263 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:13.027458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:13.027431 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:21:17.491196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:17.491164 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:17.551871 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:21:17.551835 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:21:21.978140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:21.978109 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:26.485999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:26.485963 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:31.091284 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:31.091248 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:32.322079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.322046 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp"] Nov 03 22:21:32.325867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.325850 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.382217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.382180 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp"] Nov 03 22:21:32.395590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.395564 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-dshm\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.395747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.395685 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tls-certs\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.395747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.395716 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-model-cache\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.395864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.395811 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-home\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.395913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.395858 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gbnh\" (UniqueName: \"kubernetes.io/projected/25c4c1d8-cb12-43fe-b9fe-861e31095b13-kube-api-access-4gbnh\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.396001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.395982 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tmp-dir\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.429552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.429528 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"premium-w5-f81f8f-kserve-self-signed-certs\"" Nov 03 22:21:32.459497 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.459467 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:32.497598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.497563 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tmp-dir\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.497827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.497663 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-dshm\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.497827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.497740 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tls-certs\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.497827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.497766 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-model-cache\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.497827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.497815 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-home\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.498009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.497839 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gbnh\" (UniqueName: \"kubernetes.io/projected/25c4c1d8-cb12-43fe-b9fe-861e31095b13-kube-api-access-4gbnh\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.498087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.498063 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tmp-dir\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.498176 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.498149 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-home\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.498239 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.498185 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-model-cache\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.499994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.499972 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-dshm\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.500180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.500164 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tls-certs\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.556227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.556192 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gbnh\" (UniqueName: \"kubernetes.io/projected/25c4c1d8-cb12-43fe-b9fe-861e31095b13-kube-api-access-4gbnh\") pod \"premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:32.661191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:32.661112 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:33.055959 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:21:33.055930 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25c4c1d8_cb12_43fe_b9fe_861e31095b13.slice/crio-9ffa1214c854d9f1f4c6a2b9e15ca870efbe050f1b2cc6004e3f1e7a1ae2e217 WatchSource:0}: Error finding container 9ffa1214c854d9f1f4c6a2b9e15ca870efbe050f1b2cc6004e3f1e7a1ae2e217: Status 404 returned error can't find the container with id 9ffa1214c854d9f1f4c6a2b9e15ca870efbe050f1b2cc6004e3f1e7a1ae2e217 Nov 03 22:21:33.058242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:33.058216 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp"] Nov 03 22:21:33.120635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:33.120605 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" event={"ID":"25c4c1d8-cb12-43fe-b9fe-861e31095b13","Type":"ContainerStarted","Data":"05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610"} Nov 03 22:21:33.120751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:33.120667 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" event={"ID":"25c4c1d8-cb12-43fe-b9fe-861e31095b13","Type":"ContainerStarted","Data":"9ffa1214c854d9f1f4c6a2b9e15ca870efbe050f1b2cc6004e3f1e7a1ae2e217"} Nov 03 22:21:33.238869 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:33.238811 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podStartSLOduration=1.238792469 podStartE2EDuration="1.238792469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:33.236076521 +0000 UTC m=+850.216810176" watchObservedRunningTime="2026-11-03 22:21:33.238792469 +0000 UTC m=+850.219526123" Nov 03 22:21:33.661447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:33.661414 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:35.566799 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.566768 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj"] Nov 03 22:21:35.570595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.570575 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:35.609358 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.609307 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:35.625611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.625577 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj"] Nov 03 22:21:35.630516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.630488 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjczr\" (UniqueName: \"kubernetes.io/projected/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-kube-api-access-pjczr\") pod \"maas-api-revoke-sub-37a3a810-6xdmj\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:35.630743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.630535 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:21:35.630743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.630543 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a3a810-6xdmj\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:35.630743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.630603 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a3a810-6xdmj\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:35.672595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.672562 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:21:35.732161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.732128 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjczr\" (UniqueName: \"kubernetes.io/projected/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-kube-api-access-pjczr\") pod \"maas-api-revoke-sub-37a3a810-6xdmj\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:35.732338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.732171 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a3a810-6xdmj\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:35.732338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.732220 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a3a810-6xdmj\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:35.732854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.732823 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a3a810-6xdmj\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:35.734507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.734486 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a3a810-6xdmj\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:35.789838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.789805 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjczr\" (UniqueName: \"kubernetes.io/projected/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-kube-api-access-pjczr\") pod \"maas-api-revoke-sub-37a3a810-6xdmj\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:35.884251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:35.884162 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:36.250436 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:21:36.250401 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af98e84_a10c_4119_a6e9_c9ceda7e8c96.slice/crio-9345617757f7bc792b04a13b75841ae42ff5b619e6743146864af112df0e6ae6 WatchSource:0}: Error finding container 9345617757f7bc792b04a13b75841ae42ff5b619e6743146864af112df0e6ae6: Status 404 returned error can't find the container with id 9345617757f7bc792b04a13b75841ae42ff5b619e6743146864af112df0e6ae6 Nov 03 22:21:36.253100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:36.253074 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj"] Nov 03 22:21:37.145623 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:37.145591 4298 generic.go:356] "Generic (PLEG): container finished" podID="9af98e84-a10c-4119-a6e9-c9ceda7e8c96" containerID="d81ce6e3725781af505cd07dda0127f6fefc95680f3ccf8bc99b84d5058f6a93" exitCode=0 Nov 03 22:21:37.146086 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:37.145661 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" event={"ID":"9af98e84-a10c-4119-a6e9-c9ceda7e8c96","Type":"ContainerDied","Data":"d81ce6e3725781af505cd07dda0127f6fefc95680f3ccf8bc99b84d5058f6a93"} Nov 03 22:21:37.146086 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:37.145706 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" event={"ID":"9af98e84-a10c-4119-a6e9-c9ceda7e8c96","Type":"ContainerStarted","Data":"9345617757f7bc792b04a13b75841ae42ff5b619e6743146864af112df0e6ae6"} Nov 03 22:21:38.294775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:38.294751 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:38.370710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:38.370682 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-openshift-ca\") pod \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " Nov 03 22:21:38.370910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:38.370787 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-kube-api-access-pjczr\" (UniqueName: \"kubernetes.io/projected/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-kube-api-access-pjczr\") pod \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " Nov 03 22:21:38.370910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:38.370891 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-xks-ca\") pod \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\" (UID: \"9af98e84-a10c-4119-a6e9-c9ceda7e8c96\") " Nov 03 22:21:38.371103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:38.371080 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-openshift-ca" pod "9af98e84-a10c-4119-a6e9-c9ceda7e8c96" (UID: "9af98e84-a10c-4119-a6e9-c9ceda7e8c96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:21:38.371302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:38.371282 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:21:38.373310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:38.373280 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-kube-api-access-pjczr" pod "9af98e84-a10c-4119-a6e9-c9ceda7e8c96" (UID: "9af98e84-a10c-4119-a6e9-c9ceda7e8c96"). InnerVolumeSpecName "kube-api-access-pjczr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:38.373671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:38.373623 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-xks-ca" pod "9af98e84-a10c-4119-a6e9-c9ceda7e8c96" (UID: "9af98e84-a10c-4119-a6e9-c9ceda7e8c96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:21:38.472326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:38.472284 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjczr\" (UniqueName: \"kubernetes.io/projected/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-kube-api-access-pjczr\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:21:38.472491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:38.472331 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af98e84-a10c-4119-a6e9-c9ceda7e8c96-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:21:39.156869 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:39.156829 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" event={"ID":"9af98e84-a10c-4119-a6e9-c9ceda7e8c96","Type":"ContainerDied","Data":"9345617757f7bc792b04a13b75841ae42ff5b619e6743146864af112df0e6ae6"} Nov 03 22:21:39.156869 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:39.156855 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj" Nov 03 22:21:39.157055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:39.156890 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9345617757f7bc792b04a13b75841ae42ff5b619e6743146864af112df0e6ae6" Nov 03 22:21:40.120549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:40.120515 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:42.662280 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:42.662252 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:42.674811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:42.674786 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:43.189237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:43.189206 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:21:44.577728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:44.577695 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:47.552517 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:21:47.552483 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:21:49.018299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.018261 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4"] Nov 03 22:21:49.018760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.018746 4298 cpu_manager.go:414] "containerMap: removing container" podUID="9af98e84-a10c-4119-a6e9-c9ceda7e8c96" containerName="revoke-keys" Nov 03 22:21:49.018760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.018760 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af98e84-a10c-4119-a6e9-c9ceda7e8c96" containerName="revoke-keys" Nov 03 22:21:49.018837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.018830 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af98e84-a10c-4119-a6e9-c9ceda7e8c96" containerName="revoke-keys" Nov 03 22:21:49.022142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.022127 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.045469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.045442 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:49.077723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.077697 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-41bb2b-openshift-default-dockercfg-kqcr4\"" Nov 03 22:21:49.084172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.084145 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.084265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.084186 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.084346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.084327 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtfx\" (UniqueName: \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-kube-api-access-pmtfx\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.084394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.084379 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.084427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.084419 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.084473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.084455 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.084520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.084506 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.084571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.084558 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afee2ffe-35e2-4903-849c-fb37e18b8b70-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.084614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.084602 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.134751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.134717 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4"] Nov 03 22:21:49.185783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.185745 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtfx\" (UniqueName: \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-kube-api-access-pmtfx\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.185951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.185795 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.185951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.185843 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.185951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.185869 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.185951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.185906 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.186155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.185957 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afee2ffe-35e2-4903-849c-fb37e18b8b70-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.186155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.186011 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.186155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.186057 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.186155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.186089 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.186391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.186364 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.186461 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.186397 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.186522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.186498 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.186874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.186846 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.186998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.186938 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afee2ffe-35e2-4903-849c-fb37e18b8b70-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.188093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.188070 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.188948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.188924 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.243509 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.243478 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.243851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.243828 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtfx\" (UniqueName: \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-kube-api-access-pmtfx\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.335728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.335600 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:49.750770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.750737 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4"] Nov 03 22:21:49.784862 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:21:49.784836 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafee2ffe_35e2_4903_849c_fb37e18b8b70.slice/crio-d5e23fc61dd7f56fa2c8518e172b86da7f338269f4de97f3f50e8a8b3b009c96 WatchSource:0}: Error finding container d5e23fc61dd7f56fa2c8518e172b86da7f338269f4de97f3f50e8a8b3b009c96: Status 404 returned error can't find the container with id d5e23fc61dd7f56fa2c8518e172b86da7f338269f4de97f3f50e8a8b3b009c96 Nov 03 22:21:49.787160 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.787130 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:21:49.787228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.787202 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:21:49.787263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:49.787232 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:21:50.215586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:50.215556 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" event={"ID":"afee2ffe-35e2-4903-849c-fb37e18b8b70","Type":"ContainerStarted","Data":"4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb"} Nov 03 22:21:50.216082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:50.215611 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" event={"ID":"afee2ffe-35e2-4903-849c-fb37e18b8b70","Type":"ContainerStarted","Data":"d5e23fc61dd7f56fa2c8518e172b86da7f338269f4de97f3f50e8a8b3b009c96"} Nov 03 22:21:50.335986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:50.335933 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" podStartSLOduration=2.33591801 podStartE2EDuration="2.33591801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:50.333216326 +0000 UTC m=+867.313949979" watchObservedRunningTime="2026-11-03 22:21:50.33591801 +0000 UTC m=+867.316651662" Nov 03 22:21:50.336261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:50.336241 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:50.337845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:50.337822 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:50.337946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:50.337872 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 22:21:51.345127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:51.345084 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:51.345811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:51.345775 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 22:21:52.347668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:52.346741 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:52.347668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:52.346803 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 22:21:53.339082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:53.339045 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:53.339464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:53.339438 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 22:21:53.610468 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:53.610382 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:54.336490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:54.336446 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:54.336840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:54.336805 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 22:21:55.346665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:55.346053 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:55.346665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:55.346130 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 22:21:56.336347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:56.336307 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:56.336528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:56.336378 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 22:21:57.340574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:57.340543 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:57.340976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:57.340938 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:57.341577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:57.341555 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:21:58.093963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:21:58.093930 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:01.662877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:01.662841 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:02.577778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:02.577746 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:05.465777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.465742 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd"] Nov 03 22:22:05.470966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.470947 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:05.523463 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.523427 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd"] Nov 03 22:22:05.570883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.570852 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:22:05.571033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.570917 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:22:05.598796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.598767 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42nz\" (UniqueName: \"kubernetes.io/projected/427f0a31-7fb5-46a2-b5df-f9e56817abae-kube-api-access-m42nz\") pod \"maas-api-revoke-sub-92525c10-nj5sd\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:05.598975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.598812 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92525c10-nj5sd\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:05.598975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.598935 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92525c10-nj5sd\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:05.699955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.699913 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42nz\" (UniqueName: \"kubernetes.io/projected/427f0a31-7fb5-46a2-b5df-f9e56817abae-kube-api-access-m42nz\") pod \"maas-api-revoke-sub-92525c10-nj5sd\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:05.700151 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.699971 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92525c10-nj5sd\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:05.700151 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.700056 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92525c10-nj5sd\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:05.700694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.700674 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92525c10-nj5sd\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:05.702243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.702214 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92525c10-nj5sd\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:05.757510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.757445 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42nz\" (UniqueName: \"kubernetes.io/projected/427f0a31-7fb5-46a2-b5df-f9e56817abae-kube-api-access-m42nz\") pod \"maas-api-revoke-sub-92525c10-nj5sd\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:05.784255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:05.784227 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:06.151211 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:22:06.151179 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427f0a31_7fb5_46a2_b5df_f9e56817abae.slice/crio-7d59915eb5ca715f8e7e5103e21be2a0ac674dd85f347eba922709c230172e5a WatchSource:0}: Error finding container 7d59915eb5ca715f8e7e5103e21be2a0ac674dd85f347eba922709c230172e5a: Status 404 returned error can't find the container with id 7d59915eb5ca715f8e7e5103e21be2a0ac674dd85f347eba922709c230172e5a Nov 03 22:22:06.153988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:06.153928 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd"] Nov 03 22:22:06.327751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:06.327717 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" event={"ID":"427f0a31-7fb5-46a2-b5df-f9e56817abae","Type":"ContainerStarted","Data":"7d59915eb5ca715f8e7e5103e21be2a0ac674dd85f347eba922709c230172e5a"} Nov 03 22:22:07.080062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:07.080023 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:07.334374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:07.334285 4298 generic.go:356] "Generic (PLEG): container finished" podID="427f0a31-7fb5-46a2-b5df-f9e56817abae" containerID="2e4c04b6db575ab8d0da83e52d1528b2d61a895168c988193137ba7dc3100e9a" exitCode=0 Nov 03 22:22:07.334374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:07.334331 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" event={"ID":"427f0a31-7fb5-46a2-b5df-f9e56817abae","Type":"ContainerDied","Data":"2e4c04b6db575ab8d0da83e52d1528b2d61a895168c988193137ba7dc3100e9a"} Nov 03 22:22:07.447063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:07.447035 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:08.489185 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:08.489159 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:08.633528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:08.633444 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-openshift-ca\") pod \"427f0a31-7fb5-46a2-b5df-f9e56817abae\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " Nov 03 22:22:08.633528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:08.633522 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-xks-ca\") pod \"427f0a31-7fb5-46a2-b5df-f9e56817abae\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " Nov 03 22:22:08.633768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:08.633567 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427f0a31-7fb5-46a2-b5df-f9e56817abae-kube-api-access-m42nz\" (UniqueName: \"kubernetes.io/projected/427f0a31-7fb5-46a2-b5df-f9e56817abae-kube-api-access-m42nz\") pod \"427f0a31-7fb5-46a2-b5df-f9e56817abae\" (UID: \"427f0a31-7fb5-46a2-b5df-f9e56817abae\") " Nov 03 22:22:08.633838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:08.633807 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-openshift-ca" pod "427f0a31-7fb5-46a2-b5df-f9e56817abae" (UID: "427f0a31-7fb5-46a2-b5df-f9e56817abae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:08.634121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:08.634095 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:08.635719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:08.635679 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427f0a31-7fb5-46a2-b5df-f9e56817abae-kube-api-access-m42nz" pod "427f0a31-7fb5-46a2-b5df-f9e56817abae" (UID: "427f0a31-7fb5-46a2-b5df-f9e56817abae"). InnerVolumeSpecName "kube-api-access-m42nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:08.635816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:08.635778 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-xks-ca" pod "427f0a31-7fb5-46a2-b5df-f9e56817abae" (UID: "427f0a31-7fb5-46a2-b5df-f9e56817abae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:08.735668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:08.735612 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427f0a31-7fb5-46a2-b5df-f9e56817abae-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:08.735668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:08.735664 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42nz\" (UniqueName: \"kubernetes.io/projected/427f0a31-7fb5-46a2-b5df-f9e56817abae-kube-api-access-m42nz\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:09.352885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:09.352849 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" event={"ID":"427f0a31-7fb5-46a2-b5df-f9e56817abae","Type":"ContainerDied","Data":"7d59915eb5ca715f8e7e5103e21be2a0ac674dd85f347eba922709c230172e5a"} Nov 03 22:22:09.352885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:09.352880 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd" Nov 03 22:22:09.352885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:09.352894 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d59915eb5ca715f8e7e5103e21be2a0ac674dd85f347eba922709c230172e5a" Nov 03 22:22:11.597615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:11.597580 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:12.004861 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.004826 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f"] Nov 03 22:22:12.005366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.005347 4298 cpu_manager.go:414] "containerMap: removing container" podUID="427f0a31-7fb5-46a2-b5df-f9e56817abae" containerName="revoke-keys" Nov 03 22:22:12.005464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.005368 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427f0a31-7fb5-46a2-b5df-f9e56817abae" containerName="revoke-keys" Nov 03 22:22:12.005522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.005504 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="427f0a31-7fb5-46a2-b5df-f9e56817abae" containerName="revoke-keys" Nov 03 22:22:12.008968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.008949 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.061267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.061240 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f"] Nov 03 22:22:12.110120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.110090 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"premium-w0-f61714-kserve-self-signed-certs\"" Nov 03 22:22:12.110295 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.110208 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"default-dockercfg-hrt5c\"" Nov 03 22:22:12.110295 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.110283 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"openshift-service-ca.crt\"" Nov 03 22:22:12.113365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.113347 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"kube-root-ca.crt\"" Nov 03 22:22:12.141537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.141506 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:12.173562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.173527 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr54q\" (UniqueName: \"kubernetes.io/projected/59290205-8118-46f3-9998-d49d4395b97f-kube-api-access-kr54q\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.173763 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.173592 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-home\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.173763 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.173728 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-tmp-dir\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.173846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.173794 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-model-cache\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.173891 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.173869 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-dshm\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.173944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.173909 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59290205-8118-46f3-9998-d49d4395b97f-tls-certs\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.275189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.275108 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-model-cache\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.275366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.275201 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-dshm\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.275366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.275249 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59290205-8118-46f3-9998-d49d4395b97f-tls-certs\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.275366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.275283 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr54q\" (UniqueName: \"kubernetes.io/projected/59290205-8118-46f3-9998-d49d4395b97f-kube-api-access-kr54q\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.275366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.275358 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-home\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.275584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.275448 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-tmp-dir\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.275659 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.275604 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-model-cache\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.275740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.275702 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-home\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.275808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.275761 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-tmp-dir\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.277782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.277756 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-dshm\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.277934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.277910 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59290205-8118-46f3-9998-d49d4395b97f-tls-certs\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.334483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.334438 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr54q\" (UniqueName: \"kubernetes.io/projected/59290205-8118-46f3-9998-d49d4395b97f-kube-api-access-kr54q\") pod \"premium-w0-f61714-kserve-57f4f7ffcc-72l5f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.620903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.620803 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:12.994422 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:22:12.994393 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59290205_8118_46f3_9998_d49d4395b97f.slice/crio-6474623e8c5b55dc14a560cbe43825e5f0a75545848a6280b1effa1c1206a87b WatchSource:0}: Error finding container 6474623e8c5b55dc14a560cbe43825e5f0a75545848a6280b1effa1c1206a87b: Status 404 returned error can't find the container with id 6474623e8c5b55dc14a560cbe43825e5f0a75545848a6280b1effa1c1206a87b Nov 03 22:22:12.996570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:12.996547 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f"] Nov 03 22:22:13.379434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:13.379345 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" event={"ID":"59290205-8118-46f3-9998-d49d4395b97f","Type":"ContainerStarted","Data":"6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436"} Nov 03 22:22:13.379434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:13.379396 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" event={"ID":"59290205-8118-46f3-9998-d49d4395b97f","Type":"ContainerStarted","Data":"6474623e8c5b55dc14a560cbe43825e5f0a75545848a6280b1effa1c1206a87b"} Nov 03 22:22:13.506084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:13.506008 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podStartSLOduration=2.505987101 podStartE2EDuration="2.505987101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:13.498182869 +0000 UTC m=+890.478916526" watchObservedRunningTime="2026-11-03 22:22:13.505987101 +0000 UTC m=+890.486720755" Nov 03 22:22:13.621347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:13.621307 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:15.540102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.540069 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98"] Nov 03 22:22:15.544033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.544010 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:15.600980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.600941 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98"] Nov 03 22:22:15.641723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.641692 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7e77a1a-nzk98\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:15.641909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.641887 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7e77a1a-nzk98\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:15.641957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.641922 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgchv\" (UniqueName: \"kubernetes.io/projected/afc134e6-0180-46bf-a9bf-b6136a5af0c9-kube-api-access-vgchv\") pod \"maas-api-revoke-sub-f7e77a1a-nzk98\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:15.647953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.647933 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:22:15.648084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.648065 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:22:15.742561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.742526 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7e77a1a-nzk98\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:15.742561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.742566 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgchv\" (UniqueName: \"kubernetes.io/projected/afc134e6-0180-46bf-a9bf-b6136a5af0c9-kube-api-access-vgchv\") pod \"maas-api-revoke-sub-f7e77a1a-nzk98\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:15.742797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.742635 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7e77a1a-nzk98\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:15.743366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.743335 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7e77a1a-nzk98\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:15.744937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.744917 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7e77a1a-nzk98\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:15.801199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.801127 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgchv\" (UniqueName: \"kubernetes.io/projected/afc134e6-0180-46bf-a9bf-b6136a5af0c9-kube-api-access-vgchv\") pod \"maas-api-revoke-sub-f7e77a1a-nzk98\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:15.858114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:15.858079 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:16.201909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:16.201865 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:16.224221 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:22:16.224187 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc134e6_0180_46bf_a9bf_b6136a5af0c9.slice/crio-2031282aafba88433757d5a743c58f45bc69dfa406e71c4d5c55c68e880d43de WatchSource:0}: Error finding container 2031282aafba88433757d5a743c58f45bc69dfa406e71c4d5c55c68e880d43de: Status 404 returned error can't find the container with id 2031282aafba88433757d5a743c58f45bc69dfa406e71c4d5c55c68e880d43de Nov 03 22:22:16.227139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:16.227011 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98"] Nov 03 22:22:16.396264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:16.396227 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" event={"ID":"afc134e6-0180-46bf-a9bf-b6136a5af0c9","Type":"ContainerStarted","Data":"2031282aafba88433757d5a743c58f45bc69dfa406e71c4d5c55c68e880d43de"} Nov 03 22:22:17.403351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:17.403266 4298 generic.go:356] "Generic (PLEG): container finished" podID="afc134e6-0180-46bf-a9bf-b6136a5af0c9" containerID="2e49c51d976ca66908da1f646c02a22a8fd4844d82c2a53800e93a9bce394e26" exitCode=0 Nov 03 22:22:17.403764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:17.403357 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" event={"ID":"afc134e6-0180-46bf-a9bf-b6136a5af0c9","Type":"ContainerDied","Data":"2e49c51d976ca66908da1f646c02a22a8fd4844d82c2a53800e93a9bce394e26"} Nov 03 22:22:17.552254 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:22:17.552210 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:22:18.547007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:18.546977 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:18.674487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:18.674408 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-xks-ca\") pod \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " Nov 03 22:22:18.674653 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:18.674565 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc134e6-0180-46bf-a9bf-b6136a5af0c9-kube-api-access-vgchv\" (UniqueName: \"kubernetes.io/projected/afc134e6-0180-46bf-a9bf-b6136a5af0c9-kube-api-access-vgchv\") pod \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " Nov 03 22:22:18.674712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:18.674693 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-openshift-ca\") pod \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\" (UID: \"afc134e6-0180-46bf-a9bf-b6136a5af0c9\") " Nov 03 22:22:18.675017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:18.674994 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-openshift-ca" pod "afc134e6-0180-46bf-a9bf-b6136a5af0c9" (UID: "afc134e6-0180-46bf-a9bf-b6136a5af0c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:18.676489 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:18.676465 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-xks-ca" pod "afc134e6-0180-46bf-a9bf-b6136a5af0c9" (UID: "afc134e6-0180-46bf-a9bf-b6136a5af0c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:18.676657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:18.676622 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc134e6-0180-46bf-a9bf-b6136a5af0c9-kube-api-access-vgchv" pod "afc134e6-0180-46bf-a9bf-b6136a5af0c9" (UID: "afc134e6-0180-46bf-a9bf-b6136a5af0c9"). InnerVolumeSpecName "kube-api-access-vgchv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:18.775844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:18.775813 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgchv\" (UniqueName: \"kubernetes.io/projected/afc134e6-0180-46bf-a9bf-b6136a5af0c9-kube-api-access-vgchv\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:18.775844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:18.775840 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:18.775844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:18.775850 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc134e6-0180-46bf-a9bf-b6136a5af0c9-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:19.416257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:19.416179 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" Nov 03 22:22:19.416257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:19.416205 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98" event={"ID":"afc134e6-0180-46bf-a9bf-b6136a5af0c9","Type":"ContainerDied","Data":"2031282aafba88433757d5a743c58f45bc69dfa406e71c4d5c55c68e880d43de"} Nov 03 22:22:19.416426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:19.416257 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2031282aafba88433757d5a743c58f45bc69dfa406e71c4d5c55c68e880d43de" Nov 03 22:22:20.805264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.805209 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:20.856999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.856967 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl"] Nov 03 22:22:20.857489 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.857476 4298 cpu_manager.go:414] "containerMap: removing container" podUID="afc134e6-0180-46bf-a9bf-b6136a5af0c9" containerName="revoke-keys" Nov 03 22:22:20.857543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.857490 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc134e6-0180-46bf-a9bf-b6136a5af0c9" containerName="revoke-keys" Nov 03 22:22:20.857611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.857585 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc134e6-0180-46bf-a9bf-b6136a5af0c9" containerName="revoke-keys" Nov 03 22:22:20.860968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.860950 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:20.897209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.897178 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c5bae8-qn4cl\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:20.897343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.897241 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8r6j\" (UniqueName: \"kubernetes.io/projected/ba1cb965-343f-4eeb-9aac-696a86f6df38-kube-api-access-x8r6j\") pod \"maas-api-revoke-sub-24c5bae8-qn4cl\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:20.897456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.897431 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c5bae8-qn4cl\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:20.914546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.914515 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl"] Nov 03 22:22:20.961974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.961947 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:22:20.962124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.962080 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:22:20.998877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.998842 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c5bae8-qn4cl\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:20.999036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.998913 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c5bae8-qn4cl\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:20.999036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.998974 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8r6j\" (UniqueName: \"kubernetes.io/projected/ba1cb965-343f-4eeb-9aac-696a86f6df38-kube-api-access-x8r6j\") pod \"maas-api-revoke-sub-24c5bae8-qn4cl\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:20.999809 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:20.999787 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c5bae8-qn4cl\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:21.001241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:21.001222 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c5bae8-qn4cl\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:21.056896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:21.056831 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8r6j\" (UniqueName: \"kubernetes.io/projected/ba1cb965-343f-4eeb-9aac-696a86f6df38-kube-api-access-x8r6j\") pod \"maas-api-revoke-sub-24c5bae8-qn4cl\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:21.173890 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:21.173858 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:21.541427 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:22:21.541396 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1cb965_343f_4eeb_9aac_696a86f6df38.slice/crio-ab6dbb6239fc6c7e594781e772a7d5e077551707e2e8b4df7cd3a11eef7541fe WatchSource:0}: Error finding container ab6dbb6239fc6c7e594781e772a7d5e077551707e2e8b4df7cd3a11eef7541fe: Status 404 returned error can't find the container with id ab6dbb6239fc6c7e594781e772a7d5e077551707e2e8b4df7cd3a11eef7541fe Nov 03 22:22:21.542075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:21.542015 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:21.544052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:21.544025 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl"] Nov 03 22:22:22.434286 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:22.434243 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" event={"ID":"ba1cb965-343f-4eeb-9aac-696a86f6df38","Type":"ContainerStarted","Data":"7719e1a95dbc37a50233d229402d038da9170a7c60d5cc2ff823f48d910032d4"} Nov 03 22:22:22.434726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:22.434303 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" event={"ID":"ba1cb965-343f-4eeb-9aac-696a86f6df38","Type":"ContainerStarted","Data":"ab6dbb6239fc6c7e594781e772a7d5e077551707e2e8b4df7cd3a11eef7541fe"} Nov 03 22:22:22.552041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:22.551990 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" podStartSLOduration=2.55197533 podStartE2EDuration="2.55197533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:22.549190962 +0000 UTC m=+899.529924614" watchObservedRunningTime="2026-11-03 22:22:22.55197533 +0000 UTC m=+899.532708981" Nov 03 22:22:22.621812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:22.621780 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:22.635018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:22.634992 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:23.440816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:23.440775 4298 generic.go:356] "Generic (PLEG): container finished" podID="ba1cb965-343f-4eeb-9aac-696a86f6df38" containerID="7719e1a95dbc37a50233d229402d038da9170a7c60d5cc2ff823f48d910032d4" exitCode=0 Nov 03 22:22:23.441175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:23.440859 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" event={"ID":"ba1cb965-343f-4eeb-9aac-696a86f6df38","Type":"ContainerDied","Data":"7719e1a95dbc37a50233d229402d038da9170a7c60d5cc2ff823f48d910032d4"} Nov 03 22:22:23.452569 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:23.452541 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:22:24.581524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:24.581497 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:24.634685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:24.634628 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-openshift-ca\") pod \"ba1cb965-343f-4eeb-9aac-696a86f6df38\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " Nov 03 22:22:24.634685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:24.634688 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1cb965-343f-4eeb-9aac-696a86f6df38-kube-api-access-x8r6j\" (UniqueName: \"kubernetes.io/projected/ba1cb965-343f-4eeb-9aac-696a86f6df38-kube-api-access-x8r6j\") pod \"ba1cb965-343f-4eeb-9aac-696a86f6df38\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " Nov 03 22:22:24.634888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:24.634836 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-xks-ca\") pod \"ba1cb965-343f-4eeb-9aac-696a86f6df38\" (UID: \"ba1cb965-343f-4eeb-9aac-696a86f6df38\") " Nov 03 22:22:24.635045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:24.635014 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-openshift-ca" pod "ba1cb965-343f-4eeb-9aac-696a86f6df38" (UID: "ba1cb965-343f-4eeb-9aac-696a86f6df38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:24.635209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:24.635192 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:24.636790 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:24.636765 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-xks-ca" pod "ba1cb965-343f-4eeb-9aac-696a86f6df38" (UID: "ba1cb965-343f-4eeb-9aac-696a86f6df38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:24.636969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:24.636955 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1cb965-343f-4eeb-9aac-696a86f6df38-kube-api-access-x8r6j" pod "ba1cb965-343f-4eeb-9aac-696a86f6df38" (UID: "ba1cb965-343f-4eeb-9aac-696a86f6df38"). InnerVolumeSpecName "kube-api-access-x8r6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:24.735972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:24.735893 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8r6j\" (UniqueName: \"kubernetes.io/projected/ba1cb965-343f-4eeb-9aac-696a86f6df38-kube-api-access-x8r6j\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:24.735972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:24.735924 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1cb965-343f-4eeb-9aac-696a86f6df38-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:25.396935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:25.396845 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:25.454886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:25.454850 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" event={"ID":"ba1cb965-343f-4eeb-9aac-696a86f6df38","Type":"ContainerDied","Data":"ab6dbb6239fc6c7e594781e772a7d5e077551707e2e8b4df7cd3a11eef7541fe"} Nov 03 22:22:25.454886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:25.454872 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl" Nov 03 22:22:25.455094 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:25.454902 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab6dbb6239fc6c7e594781e772a7d5e077551707e2e8b4df7cd3a11eef7541fe" Nov 03 22:22:30.016479 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:30.016441 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:34.490207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:34.490173 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:35.565431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.565396 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9"] Nov 03 22:22:35.565940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.565883 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ba1cb965-343f-4eeb-9aac-696a86f6df38" containerName="revoke-keys" Nov 03 22:22:35.565940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.565894 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1cb965-343f-4eeb-9aac-696a86f6df38" containerName="revoke-keys" Nov 03 22:22:35.566053 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.565980 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1cb965-343f-4eeb-9aac-696a86f6df38" containerName="revoke-keys" Nov 03 22:22:35.569501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.569483 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:35.622535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.622503 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9"] Nov 03 22:22:35.654860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.654820 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf616b2-vvgc9\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:35.655037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.654873 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwl85\" (UniqueName: \"kubernetes.io/projected/1771fe6d-7213-4e5d-8875-311183598d2e-kube-api-access-pwl85\") pod \"maas-api-revoke-sub-fbf616b2-vvgc9\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:35.655096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.655050 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf616b2-vvgc9\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:35.670314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.670279 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:22:35.670480 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.670368 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:22:35.756882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.756850 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf616b2-vvgc9\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:35.757046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.756893 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwl85\" (UniqueName: \"kubernetes.io/projected/1771fe6d-7213-4e5d-8875-311183598d2e-kube-api-access-pwl85\") pod \"maas-api-revoke-sub-fbf616b2-vvgc9\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:35.757046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.756943 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf616b2-vvgc9\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:35.757631 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.757605 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf616b2-vvgc9\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:35.759263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.759239 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf616b2-vvgc9\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:35.815067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.815039 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwl85\" (UniqueName: \"kubernetes.io/projected/1771fe6d-7213-4e5d-8875-311183598d2e-kube-api-access-pwl85\") pod \"maas-api-revoke-sub-fbf616b2-vvgc9\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:35.882311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:35.882233 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:36.253072 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:22:36.253032 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1771fe6d_7213_4e5d_8875_311183598d2e.slice/crio-30de9d45b39e45e60b736068399e2b5e4e23b989cf0c674ddc202b83950314d0 WatchSource:0}: Error finding container 30de9d45b39e45e60b736068399e2b5e4e23b989cf0c674ddc202b83950314d0: Status 404 returned error can't find the container with id 30de9d45b39e45e60b736068399e2b5e4e23b989cf0c674ddc202b83950314d0 Nov 03 22:22:36.255943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:36.255918 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9"] Nov 03 22:22:36.513723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:36.513620 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" event={"ID":"1771fe6d-7213-4e5d-8875-311183598d2e","Type":"ContainerStarted","Data":"30de9d45b39e45e60b736068399e2b5e4e23b989cf0c674ddc202b83950314d0"} Nov 03 22:22:37.521030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:37.520973 4298 generic.go:356] "Generic (PLEG): container finished" podID="1771fe6d-7213-4e5d-8875-311183598d2e" containerID="e8a5274b4a9f825bc51cbe162877828d089c3a10812e1a467fc67c8e474ae503" exitCode=0 Nov 03 22:22:37.521481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:37.521065 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" event={"ID":"1771fe6d-7213-4e5d-8875-311183598d2e","Type":"ContainerDied","Data":"e8a5274b4a9f825bc51cbe162877828d089c3a10812e1a467fc67c8e474ae503"} Nov 03 22:22:38.670162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:38.670136 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:38.789211 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:38.789126 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1771fe6d-7213-4e5d-8875-311183598d2e-kube-api-access-pwl85\" (UniqueName: \"kubernetes.io/projected/1771fe6d-7213-4e5d-8875-311183598d2e-kube-api-access-pwl85\") pod \"1771fe6d-7213-4e5d-8875-311183598d2e\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " Nov 03 22:22:38.789368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:38.789311 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-openshift-ca\") pod \"1771fe6d-7213-4e5d-8875-311183598d2e\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " Nov 03 22:22:38.789774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:38.789431 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-xks-ca\") pod \"1771fe6d-7213-4e5d-8875-311183598d2e\" (UID: \"1771fe6d-7213-4e5d-8875-311183598d2e\") " Nov 03 22:22:38.789774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:38.789734 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-openshift-ca" pod "1771fe6d-7213-4e5d-8875-311183598d2e" (UID: "1771fe6d-7213-4e5d-8875-311183598d2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:38.790004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:38.789854 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:38.792247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:38.792220 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1771fe6d-7213-4e5d-8875-311183598d2e-kube-api-access-pwl85" pod "1771fe6d-7213-4e5d-8875-311183598d2e" (UID: "1771fe6d-7213-4e5d-8875-311183598d2e"). InnerVolumeSpecName "kube-api-access-pwl85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:38.792405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:38.792383 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-xks-ca" pod "1771fe6d-7213-4e5d-8875-311183598d2e" (UID: "1771fe6d-7213-4e5d-8875-311183598d2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:38.891311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:38.891282 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1771fe6d-7213-4e5d-8875-311183598d2e-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:38.891311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:38.891309 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwl85\" (UniqueName: \"kubernetes.io/projected/1771fe6d-7213-4e5d-8875-311183598d2e-kube-api-access-pwl85\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:38.987893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:38.987861 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:39.533340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:39.533316 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" Nov 03 22:22:39.533513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:39.533315 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9" event={"ID":"1771fe6d-7213-4e5d-8875-311183598d2e","Type":"ContainerDied","Data":"30de9d45b39e45e60b736068399e2b5e4e23b989cf0c674ddc202b83950314d0"} Nov 03 22:22:39.533513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:39.533439 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30de9d45b39e45e60b736068399e2b5e4e23b989cf0c674ddc202b83950314d0" Nov 03 22:22:43.478165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:43.478136 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:46.324185 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.324151 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv"] Nov 03 22:22:46.324633 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.324619 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1771fe6d-7213-4e5d-8875-311183598d2e" containerName="revoke-keys" Nov 03 22:22:46.324693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.324634 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1771fe6d-7213-4e5d-8875-311183598d2e" containerName="revoke-keys" Nov 03 22:22:46.324768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.324758 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1771fe6d-7213-4e5d-8875-311183598d2e" containerName="revoke-keys" Nov 03 22:22:46.329048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.329033 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:46.369259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.369225 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6384fbd5-29mcv\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:46.369404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.369308 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6384fbd5-29mcv\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:46.369404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.369377 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxcf9\" (UniqueName: \"kubernetes.io/projected/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-kube-api-access-hxcf9\") pod \"maas-api-revoke-sub-6384fbd5-29mcv\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:46.382304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.382272 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv"] Nov 03 22:22:46.430264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.430230 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:22:46.430423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.430402 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:22:46.470752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.470720 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6384fbd5-29mcv\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:46.470916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.470817 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6384fbd5-29mcv\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:46.470916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.470855 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxcf9\" (UniqueName: \"kubernetes.io/projected/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-kube-api-access-hxcf9\") pod \"maas-api-revoke-sub-6384fbd5-29mcv\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:46.471370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.471351 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6384fbd5-29mcv\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:46.473166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.473146 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6384fbd5-29mcv\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:46.528352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.528324 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxcf9\" (UniqueName: \"kubernetes.io/projected/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-kube-api-access-hxcf9\") pod \"maas-api-revoke-sub-6384fbd5-29mcv\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:46.642054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.641958 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:46.870847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:46.870809 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:47.009149 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:22:47.009117 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8444b5_9469_4f90_9f7b_6d2d44d69c75.slice/crio-0813d54089b82d4f6df84d177109192409b0931c08085763813789d6842180b2 WatchSource:0}: Error finding container 0813d54089b82d4f6df84d177109192409b0931c08085763813789d6842180b2: Status 404 returned error can't find the container with id 0813d54089b82d4f6df84d177109192409b0931c08085763813789d6842180b2 Nov 03 22:22:47.012143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:47.012118 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv"] Nov 03 22:22:47.551889 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:22:47.551847 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:22:47.577888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:47.577857 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" event={"ID":"cc8444b5-9469-4f90-9f7b-6d2d44d69c75","Type":"ContainerStarted","Data":"a232adaf186bb49ba9bc3c8a7396bd12c5f62dba9747311a759ee4b7fc5082ea"} Nov 03 22:22:47.578085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:47.577909 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" event={"ID":"cc8444b5-9469-4f90-9f7b-6d2d44d69c75","Type":"ContainerStarted","Data":"0813d54089b82d4f6df84d177109192409b0931c08085763813789d6842180b2"} Nov 03 22:22:47.693085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:47.692986 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" podStartSLOduration=1.692973214 podStartE2EDuration="1.692973214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:47.692576537 +0000 UTC m=+924.673310190" watchObservedRunningTime="2026-11-03 22:22:47.692973214 +0000 UTC m=+924.673706891" Nov 03 22:22:47.995627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:47.995585 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:48.584729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:48.584693 4298 generic.go:356] "Generic (PLEG): container finished" podID="cc8444b5-9469-4f90-9f7b-6d2d44d69c75" containerID="a232adaf186bb49ba9bc3c8a7396bd12c5f62dba9747311a759ee4b7fc5082ea" exitCode=0 Nov 03 22:22:48.585155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:48.584751 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" event={"ID":"cc8444b5-9469-4f90-9f7b-6d2d44d69c75","Type":"ContainerDied","Data":"a232adaf186bb49ba9bc3c8a7396bd12c5f62dba9747311a759ee4b7fc5082ea"} Nov 03 22:22:49.724400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:49.724374 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:49.816994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:49.816959 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-xks-ca\") pod \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " Nov 03 22:22:49.817150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:49.817031 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-openshift-ca\") pod \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " Nov 03 22:22:49.817150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:49.817140 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-kube-api-access-hxcf9\" (UniqueName: \"kubernetes.io/projected/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-kube-api-access-hxcf9\") pod \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\" (UID: \"cc8444b5-9469-4f90-9f7b-6d2d44d69c75\") " Nov 03 22:22:49.817401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:49.817377 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-openshift-ca" pod "cc8444b5-9469-4f90-9f7b-6d2d44d69c75" (UID: "cc8444b5-9469-4f90-9f7b-6d2d44d69c75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:49.817543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:49.817528 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:49.819186 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:49.819158 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-xks-ca" pod "cc8444b5-9469-4f90-9f7b-6d2d44d69c75" (UID: "cc8444b5-9469-4f90-9f7b-6d2d44d69c75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:49.819186 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:49.819170 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-kube-api-access-hxcf9" pod "cc8444b5-9469-4f90-9f7b-6d2d44d69c75" (UID: "cc8444b5-9469-4f90-9f7b-6d2d44d69c75"). InnerVolumeSpecName "kube-api-access-hxcf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:49.919188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:49.919111 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:49.919188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:49.919140 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxcf9\" (UniqueName: \"kubernetes.io/projected/cc8444b5-9469-4f90-9f7b-6d2d44d69c75-kube-api-access-hxcf9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:22:50.596586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:50.596556 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" Nov 03 22:22:50.596781 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:50.596585 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv" event={"ID":"cc8444b5-9469-4f90-9f7b-6d2d44d69c75","Type":"ContainerDied","Data":"0813d54089b82d4f6df84d177109192409b0931c08085763813789d6842180b2"} Nov 03 22:22:50.596781 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:50.596666 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0813d54089b82d4f6df84d177109192409b0931c08085763813789d6842180b2" Nov 03 22:22:52.699192 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:52.699152 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:57.188203 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:57.188162 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:58.785656 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:58.785614 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2"] Nov 03 22:22:58.786372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:58.786352 4298 cpu_manager.go:414] "containerMap: removing container" podUID="cc8444b5-9469-4f90-9f7b-6d2d44d69c75" containerName="revoke-keys" Nov 03 22:22:58.786415 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:58.786375 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8444b5-9469-4f90-9f7b-6d2d44d69c75" containerName="revoke-keys" Nov 03 22:22:58.786566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:58.786553 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8444b5-9469-4f90-9f7b-6d2d44d69c75" containerName="revoke-keys" Nov 03 22:22:58.790414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:58.790396 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:22:58.842596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:58.842560 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2"] Nov 03 22:22:58.890158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:58.890103 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:22:58.890158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:58.890129 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:22:58.926715 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:58.926682 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f4d8d7b-zc7w2\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:22:58.926715 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:58.926715 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52j2g\" (UniqueName: \"kubernetes.io/projected/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-kube-api-access-52j2g\") pod \"maas-api-revoke-sub-2f4d8d7b-zc7w2\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:22:58.926960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:58.926822 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f4d8d7b-zc7w2\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:22:59.028045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:59.028012 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f4d8d7b-zc7w2\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:22:59.028045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:59.028049 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52j2g\" (UniqueName: \"kubernetes.io/projected/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-kube-api-access-52j2g\") pod \"maas-api-revoke-sub-2f4d8d7b-zc7w2\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:22:59.028289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:59.028112 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f4d8d7b-zc7w2\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:22:59.029001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:59.028972 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f4d8d7b-zc7w2\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:22:59.030765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:59.030735 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f4d8d7b-zc7w2\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:22:59.086258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:59.086177 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52j2g\" (UniqueName: \"kubernetes.io/projected/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-kube-api-access-52j2g\") pod \"maas-api-revoke-sub-2f4d8d7b-zc7w2\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:22:59.104829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:59.104796 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:22:59.472655 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:22:59.472604 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b92ffa9_fc55_4933_a64f_9348bfd8e18a.slice/crio-8967b586c737c6c7a44d2e2dfdf217e032e35043e98d92e964e1a502a0e3f36b WatchSource:0}: Error finding container 8967b586c737c6c7a44d2e2dfdf217e032e35043e98d92e964e1a502a0e3f36b: Status 404 returned error can't find the container with id 8967b586c737c6c7a44d2e2dfdf217e032e35043e98d92e964e1a502a0e3f36b Nov 03 22:22:59.476003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:59.475980 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2"] Nov 03 22:22:59.659660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:22:59.659602 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" event={"ID":"4b92ffa9-fc55-4933-a64f-9348bfd8e18a","Type":"ContainerStarted","Data":"8967b586c737c6c7a44d2e2dfdf217e032e35043e98d92e964e1a502a0e3f36b"} Nov 03 22:23:00.667806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:00.667719 4298 generic.go:356] "Generic (PLEG): container finished" podID="4b92ffa9-fc55-4933-a64f-9348bfd8e18a" containerID="5db33207d28f5960862ecdf5f275f3e28c78feaef75f29cd6858f8b3e563abb2" exitCode=0 Nov 03 22:23:00.667806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:00.667780 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" event={"ID":"4b92ffa9-fc55-4933-a64f-9348bfd8e18a","Type":"ContainerDied","Data":"5db33207d28f5960862ecdf5f275f3e28c78feaef75f29cd6858f8b3e563abb2"} Nov 03 22:23:01.788811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:01.788767 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:01.834144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:01.834118 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:23:01.961656 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:01.961607 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-kube-api-access-52j2g\" (UniqueName: \"kubernetes.io/projected/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-kube-api-access-52j2g\") pod \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " Nov 03 22:23:01.961856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:01.961698 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-xks-ca\") pod \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " Nov 03 22:23:01.961856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:01.961738 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-openshift-ca\") pod \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\" (UID: \"4b92ffa9-fc55-4933-a64f-9348bfd8e18a\") " Nov 03 22:23:01.962134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:01.962110 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-openshift-ca" pod "4b92ffa9-fc55-4933-a64f-9348bfd8e18a" (UID: "4b92ffa9-fc55-4933-a64f-9348bfd8e18a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:01.962251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:01.962237 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:01.964219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:01.964197 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-kube-api-access-52j2g" pod "4b92ffa9-fc55-4933-a64f-9348bfd8e18a" (UID: "4b92ffa9-fc55-4933-a64f-9348bfd8e18a"). InnerVolumeSpecName "kube-api-access-52j2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:01.966743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:01.966717 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-xks-ca" pod "4b92ffa9-fc55-4933-a64f-9348bfd8e18a" (UID: "4b92ffa9-fc55-4933-a64f-9348bfd8e18a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:02.063741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:02.063696 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52j2g\" (UniqueName: \"kubernetes.io/projected/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-kube-api-access-52j2g\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.063741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:02.063733 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b92ffa9-fc55-4933-a64f-9348bfd8e18a-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.680010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:02.679981 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" Nov 03 22:23:02.680189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:02.679977 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2" event={"ID":"4b92ffa9-fc55-4933-a64f-9348bfd8e18a","Type":"ContainerDied","Data":"8967b586c737c6c7a44d2e2dfdf217e032e35043e98d92e964e1a502a0e3f36b"} Nov 03 22:23:02.680189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:02.680118 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8967b586c737c6c7a44d2e2dfdf217e032e35043e98d92e964e1a502a0e3f36b" Nov 03 22:23:03.163520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.163485 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb"] Nov 03 22:23:03.164000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.163987 4298 cpu_manager.go:414] "containerMap: removing container" podUID="4b92ffa9-fc55-4933-a64f-9348bfd8e18a" containerName="revoke-keys" Nov 03 22:23:03.164057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.164001 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b92ffa9-fc55-4933-a64f-9348bfd8e18a" containerName="revoke-keys" Nov 03 22:23:03.164104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.164095 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b92ffa9-fc55-4933-a64f-9348bfd8e18a" containerName="revoke-keys" Nov 03 22:23:03.167414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.167395 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:03.220694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.220632 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:23:03.220889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.220752 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:23:03.221202 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.221179 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb"] Nov 03 22:23:03.278223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.278186 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4171f1ab-7gkzb\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:03.278407 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.278250 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4171f1ab-7gkzb\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:03.278407 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.278352 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cvr9\" (UniqueName: \"kubernetes.io/projected/1b9d9c7d-0245-4063-8d06-5b2384bc538d-kube-api-access-7cvr9\") pod \"maas-api-revoke-sub-4171f1ab-7gkzb\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:03.379207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.379167 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4171f1ab-7gkzb\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:03.379388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.379235 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4171f1ab-7gkzb\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:03.379388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.379288 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cvr9\" (UniqueName: \"kubernetes.io/projected/1b9d9c7d-0245-4063-8d06-5b2384bc538d-kube-api-access-7cvr9\") pod \"maas-api-revoke-sub-4171f1ab-7gkzb\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:03.379980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.379955 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4171f1ab-7gkzb\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:03.381745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.381726 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4171f1ab-7gkzb\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:03.439313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.439237 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cvr9\" (UniqueName: \"kubernetes.io/projected/1b9d9c7d-0245-4063-8d06-5b2384bc538d-kube-api-access-7cvr9\") pod \"maas-api-revoke-sub-4171f1ab-7gkzb\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:03.480338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.480296 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:03.847609 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:23:03.847577 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9d9c7d_0245_4063_8d06_5b2384bc538d.slice/crio-8f47050d36b3cb8db561a8e1da4f775ffa3e5ca88e5d2a1c3dcab11e06126ad4 WatchSource:0}: Error finding container 8f47050d36b3cb8db561a8e1da4f775ffa3e5ca88e5d2a1c3dcab11e06126ad4: Status 404 returned error can't find the container with id 8f47050d36b3cb8db561a8e1da4f775ffa3e5ca88e5d2a1c3dcab11e06126ad4 Nov 03 22:23:03.850320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:03.850294 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb"] Nov 03 22:23:04.692812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:04.692779 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" event={"ID":"1b9d9c7d-0245-4063-8d06-5b2384bc538d","Type":"ContainerStarted","Data":"4470d055c2e43db974128524efd969ca7a97f6a216848131eec5f64592b7f1e8"} Nov 03 22:23:04.693140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:04.692844 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" event={"ID":"1b9d9c7d-0245-4063-8d06-5b2384bc538d","Type":"ContainerStarted","Data":"8f47050d36b3cb8db561a8e1da4f775ffa3e5ca88e5d2a1c3dcab11e06126ad4"} Nov 03 22:23:04.810700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:04.810620 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" podStartSLOduration=1.8106062010000001 podStartE2EDuration="1.810606201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:04.807461303 +0000 UTC m=+941.788194957" watchObservedRunningTime="2026-11-03 22:23:04.810606201 +0000 UTC m=+941.791339854" Nov 03 22:23:05.707299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:05.707264 4298 generic.go:356] "Generic (PLEG): container finished" podID="1b9d9c7d-0245-4063-8d06-5b2384bc538d" containerID="4470d055c2e43db974128524efd969ca7a97f6a216848131eec5f64592b7f1e8" exitCode=0 Nov 03 22:23:05.707749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:05.707343 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" event={"ID":"1b9d9c7d-0245-4063-8d06-5b2384bc538d","Type":"ContainerDied","Data":"4470d055c2e43db974128524efd969ca7a97f6a216848131eec5f64592b7f1e8"} Nov 03 22:23:06.286004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:06.285975 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:06.855302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:06.855274 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:07.020468 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.020383 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9d9c7d-0245-4063-8d06-5b2384bc538d-kube-api-access-7cvr9\" (UniqueName: \"kubernetes.io/projected/1b9d9c7d-0245-4063-8d06-5b2384bc538d-kube-api-access-7cvr9\") pod \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " Nov 03 22:23:07.020468 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.020448 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-xks-ca\") pod \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " Nov 03 22:23:07.020714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.020488 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-openshift-ca\") pod \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\" (UID: \"1b9d9c7d-0245-4063-8d06-5b2384bc538d\") " Nov 03 22:23:07.020898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.020876 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-openshift-ca" pod "1b9d9c7d-0245-4063-8d06-5b2384bc538d" (UID: "1b9d9c7d-0245-4063-8d06-5b2384bc538d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:07.022509 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.022485 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-xks-ca" pod "1b9d9c7d-0245-4063-8d06-5b2384bc538d" (UID: "1b9d9c7d-0245-4063-8d06-5b2384bc538d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:07.022569 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.022531 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9d9c7d-0245-4063-8d06-5b2384bc538d-kube-api-access-7cvr9" pod "1b9d9c7d-0245-4063-8d06-5b2384bc538d" (UID: "1b9d9c7d-0245-4063-8d06-5b2384bc538d"). InnerVolumeSpecName "kube-api-access-7cvr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:07.122088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.122053 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cvr9\" (UniqueName: \"kubernetes.io/projected/1b9d9c7d-0245-4063-8d06-5b2384bc538d-kube-api-access-7cvr9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:07.122088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.122082 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:07.122088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.122094 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d9c7d-0245-4063-8d06-5b2384bc538d-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:07.719293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.719262 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" event={"ID":"1b9d9c7d-0245-4063-8d06-5b2384bc538d","Type":"ContainerDied","Data":"8f47050d36b3cb8db561a8e1da4f775ffa3e5ca88e5d2a1c3dcab11e06126ad4"} Nov 03 22:23:07.719470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.719308 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f47050d36b3cb8db561a8e1da4f775ffa3e5ca88e5d2a1c3dcab11e06126ad4" Nov 03 22:23:07.719470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.719275 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb" Nov 03 22:23:07.794989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:07.794946 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:10.731410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:10.731370 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:15.018699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.018661 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f"] Nov 03 22:23:15.019358 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.019342 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1b9d9c7d-0245-4063-8d06-5b2384bc538d" containerName="revoke-keys" Nov 03 22:23:15.019426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.019359 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9d9c7d-0245-4063-8d06-5b2384bc538d" containerName="revoke-keys" Nov 03 22:23:15.019500 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.019489 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9d9c7d-0245-4063-8d06-5b2384bc538d" containerName="revoke-keys" Nov 03 22:23:15.023306 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.023284 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:15.075449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.075417 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f"] Nov 03 22:23:15.102889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.102859 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c8eeaf-ncr4f\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:15.103045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.102926 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c8eeaf-ncr4f\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:15.103091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.103042 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fjq2\" (UniqueName: \"kubernetes.io/projected/cf16452f-e7d7-4e81-b851-0c9269393e60-kube-api-access-2fjq2\") pod \"maas-api-revoke-sub-c7c8eeaf-ncr4f\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:15.122637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.122612 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:23:15.122799 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.122752 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:23:15.204050 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.204013 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c8eeaf-ncr4f\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:15.204272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.204104 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fjq2\" (UniqueName: \"kubernetes.io/projected/cf16452f-e7d7-4e81-b851-0c9269393e60-kube-api-access-2fjq2\") pod \"maas-api-revoke-sub-c7c8eeaf-ncr4f\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:15.204385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.204351 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c8eeaf-ncr4f\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:15.205461 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.205410 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c8eeaf-ncr4f\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:15.207472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.207442 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c8eeaf-ncr4f\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:15.216909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.216874 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:15.262233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.262205 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fjq2\" (UniqueName: \"kubernetes.io/projected/cf16452f-e7d7-4e81-b851-0c9269393e60-kube-api-access-2fjq2\") pod \"maas-api-revoke-sub-c7c8eeaf-ncr4f\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:15.339475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.339395 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:15.706135 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:23:15.706105 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf16452f_e7d7_4e81_b851_0c9269393e60.slice/crio-7e87a156a20a6a5e9c62a1707c36ffb8d72423b6b9100da5a0a749f31b1a5797 WatchSource:0}: Error finding container 7e87a156a20a6a5e9c62a1707c36ffb8d72423b6b9100da5a0a749f31b1a5797: Status 404 returned error can't find the container with id 7e87a156a20a6a5e9c62a1707c36ffb8d72423b6b9100da5a0a749f31b1a5797 Nov 03 22:23:15.708741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.708718 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f"] Nov 03 22:23:15.769232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:15.769195 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" event={"ID":"cf16452f-e7d7-4e81-b851-0c9269393e60","Type":"ContainerStarted","Data":"7e87a156a20a6a5e9c62a1707c36ffb8d72423b6b9100da5a0a749f31b1a5797"} Nov 03 22:23:16.777556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:16.777469 4298 generic.go:356] "Generic (PLEG): container finished" podID="cf16452f-e7d7-4e81-b851-0c9269393e60" containerID="04c504c40e72613c137665178ebf4cb8345d6ba882614037f721840274a39406" exitCode=0 Nov 03 22:23:16.777952 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:16.777559 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" event={"ID":"cf16452f-e7d7-4e81-b851-0c9269393e60","Type":"ContainerDied","Data":"04c504c40e72613c137665178ebf4cb8345d6ba882614037f721840274a39406"} Nov 03 22:23:17.552084 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:23:17.552051 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:23:17.940518 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:17.940491 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:18.035140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.035109 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-xks-ca\") pod \"cf16452f-e7d7-4e81-b851-0c9269393e60\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " Nov 03 22:23:18.035287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.035211 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-openshift-ca\") pod \"cf16452f-e7d7-4e81-b851-0c9269393e60\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " Nov 03 22:23:18.035287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.035253 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf16452f-e7d7-4e81-b851-0c9269393e60-kube-api-access-2fjq2\" (UniqueName: \"kubernetes.io/projected/cf16452f-e7d7-4e81-b851-0c9269393e60-kube-api-access-2fjq2\") pod \"cf16452f-e7d7-4e81-b851-0c9269393e60\" (UID: \"cf16452f-e7d7-4e81-b851-0c9269393e60\") " Nov 03 22:23:18.035594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.035564 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-openshift-ca" pod "cf16452f-e7d7-4e81-b851-0c9269393e60" (UID: "cf16452f-e7d7-4e81-b851-0c9269393e60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:18.037213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.037195 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-xks-ca" pod "cf16452f-e7d7-4e81-b851-0c9269393e60" (UID: "cf16452f-e7d7-4e81-b851-0c9269393e60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:18.037359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.037338 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf16452f-e7d7-4e81-b851-0c9269393e60-kube-api-access-2fjq2" pod "cf16452f-e7d7-4e81-b851-0c9269393e60" (UID: "cf16452f-e7d7-4e81-b851-0c9269393e60"). InnerVolumeSpecName "kube-api-access-2fjq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:18.137065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.136987 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:18.137065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.137016 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf16452f-e7d7-4e81-b851-0c9269393e60-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:18.137065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.137027 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fjq2\" (UniqueName: \"kubernetes.io/projected/cf16452f-e7d7-4e81-b851-0c9269393e60-kube-api-access-2fjq2\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:18.797401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.797362 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" event={"ID":"cf16452f-e7d7-4e81-b851-0c9269393e60","Type":"ContainerDied","Data":"7e87a156a20a6a5e9c62a1707c36ffb8d72423b6b9100da5a0a749f31b1a5797"} Nov 03 22:23:18.797401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.797397 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f" Nov 03 22:23:18.797622 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:18.797424 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e87a156a20a6a5e9c62a1707c36ffb8d72423b6b9100da5a0a749f31b1a5797" Nov 03 22:23:19.625435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.625403 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6"] Nov 03 22:23:19.625904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.625890 4298 cpu_manager.go:414] "containerMap: removing container" podUID="cf16452f-e7d7-4e81-b851-0c9269393e60" containerName="revoke-keys" Nov 03 22:23:19.625948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.625905 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf16452f-e7d7-4e81-b851-0c9269393e60" containerName="revoke-keys" Nov 03 22:23:19.626005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.625996 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf16452f-e7d7-4e81-b851-0c9269393e60" containerName="revoke-keys" Nov 03 22:23:19.629681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.629658 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:19.650445 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.650415 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqskr\" (UniqueName: \"kubernetes.io/projected/3970029b-1835-4668-be00-23fdd5511ce3-kube-api-access-vqskr\") pod \"maas-api-revoke-sub-3444cacd-kfnr6\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:19.650562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.650463 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3444cacd-kfnr6\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:19.650675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.650632 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3444cacd-kfnr6\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:19.684265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.684224 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:19.688402 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.688374 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:23:19.696349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.696272 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6"] Nov 03 22:23:19.736385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.736356 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:23:19.751341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.751316 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqskr\" (UniqueName: \"kubernetes.io/projected/3970029b-1835-4668-be00-23fdd5511ce3-kube-api-access-vqskr\") pod \"maas-api-revoke-sub-3444cacd-kfnr6\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:19.751501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.751346 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3444cacd-kfnr6\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:19.751501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.751409 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3444cacd-kfnr6\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:19.752146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.752124 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3444cacd-kfnr6\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:19.753929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.753906 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3444cacd-kfnr6\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:19.809279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.809252 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqskr\" (UniqueName: \"kubernetes.io/projected/3970029b-1835-4668-be00-23fdd5511ce3-kube-api-access-vqskr\") pod \"maas-api-revoke-sub-3444cacd-kfnr6\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:19.944797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:19.944719 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:20.309104 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:23:20.309076 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3970029b_1835_4668_be00_23fdd5511ce3.slice/crio-cf1986e5e379d9e369c646d4495f8f6d2aad9c45cbc1288a81bccb2fbe92fa6c WatchSource:0}: Error finding container cf1986e5e379d9e369c646d4495f8f6d2aad9c45cbc1288a81bccb2fbe92fa6c: Status 404 returned error can't find the container with id cf1986e5e379d9e369c646d4495f8f6d2aad9c45cbc1288a81bccb2fbe92fa6c Nov 03 22:23:20.312372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:20.312349 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6"] Nov 03 22:23:20.811154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:20.811112 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" event={"ID":"3970029b-1835-4668-be00-23fdd5511ce3","Type":"ContainerStarted","Data":"a3426280a127f5974d1366e5812c1569743b9beeee81a12c036f06be33eabb28"} Nov 03 22:23:20.811531 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:20.811180 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" event={"ID":"3970029b-1835-4668-be00-23fdd5511ce3","Type":"ContainerStarted","Data":"cf1986e5e379d9e369c646d4495f8f6d2aad9c45cbc1288a81bccb2fbe92fa6c"} Nov 03 22:23:20.926231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:20.926138 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" podStartSLOduration=1.926124081 podStartE2EDuration="1.926124081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:20.925928359 +0000 UTC m=+957.906662011" watchObservedRunningTime="2026-11-03 22:23:20.926124081 +0000 UTC m=+957.906857733" Nov 03 22:23:21.736348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:21.736316 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:21.820362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:21.820323 4298 generic.go:356] "Generic (PLEG): container finished" podID="3970029b-1835-4668-be00-23fdd5511ce3" containerID="a3426280a127f5974d1366e5812c1569743b9beeee81a12c036f06be33eabb28" exitCode=0 Nov 03 22:23:21.820885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:21.820421 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" event={"ID":"3970029b-1835-4668-be00-23fdd5511ce3","Type":"ContainerDied","Data":"a3426280a127f5974d1366e5812c1569743b9beeee81a12c036f06be33eabb28"} Nov 03 22:23:22.958892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:22.958865 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:23.018626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.018597 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-xks-ca\") pod \"3970029b-1835-4668-be00-23fdd5511ce3\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " Nov 03 22:23:23.018794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.018703 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-openshift-ca\") pod \"3970029b-1835-4668-be00-23fdd5511ce3\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " Nov 03 22:23:23.018883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.018851 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3970029b-1835-4668-be00-23fdd5511ce3-kube-api-access-vqskr\" (UniqueName: \"kubernetes.io/projected/3970029b-1835-4668-be00-23fdd5511ce3-kube-api-access-vqskr\") pod \"3970029b-1835-4668-be00-23fdd5511ce3\" (UID: \"3970029b-1835-4668-be00-23fdd5511ce3\") " Nov 03 22:23:23.019092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.019065 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-openshift-ca" pod "3970029b-1835-4668-be00-23fdd5511ce3" (UID: "3970029b-1835-4668-be00-23fdd5511ce3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:23.019296 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.019277 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:23.020867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.020846 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3970029b-1835-4668-be00-23fdd5511ce3-kube-api-access-vqskr" pod "3970029b-1835-4668-be00-23fdd5511ce3" (UID: "3970029b-1835-4668-be00-23fdd5511ce3"). InnerVolumeSpecName "kube-api-access-vqskr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:23.020974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.020937 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-xks-ca" pod "3970029b-1835-4668-be00-23fdd5511ce3" (UID: "3970029b-1835-4668-be00-23fdd5511ce3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:23.120374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.120291 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqskr\" (UniqueName: \"kubernetes.io/projected/3970029b-1835-4668-be00-23fdd5511ce3-kube-api-access-vqskr\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:23.120374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.120331 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3970029b-1835-4668-be00-23fdd5511ce3-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:23.832493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.832458 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" Nov 03 22:23:23.832493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.832476 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6" event={"ID":"3970029b-1835-4668-be00-23fdd5511ce3","Type":"ContainerDied","Data":"cf1986e5e379d9e369c646d4495f8f6d2aad9c45cbc1288a81bccb2fbe92fa6c"} Nov 03 22:23:23.832727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:23.832524 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf1986e5e379d9e369c646d4495f8f6d2aad9c45cbc1288a81bccb2fbe92fa6c" Nov 03 22:23:24.165472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:24.165388 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:28.798546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:28.798475 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:30.380472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.380438 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x"] Nov 03 22:23:30.380932 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.380919 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3970029b-1835-4668-be00-23fdd5511ce3" containerName="revoke-keys" Nov 03 22:23:30.380988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.380933 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3970029b-1835-4668-be00-23fdd5511ce3" containerName="revoke-keys" Nov 03 22:23:30.381035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.381026 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3970029b-1835-4668-be00-23fdd5511ce3" containerName="revoke-keys" Nov 03 22:23:30.384332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.384298 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:30.439338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.439302 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x"] Nov 03 22:23:30.485666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.485609 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:23:30.485819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.485752 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:23:30.497877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.497847 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blwr9\" (UniqueName: \"kubernetes.io/projected/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-kube-api-access-blwr9\") pod \"maas-api-revoke-sub-e5a70ecf-cmk4x\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:30.498017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.497897 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a70ecf-cmk4x\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:30.498017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.497991 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a70ecf-cmk4x\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:30.598755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.598723 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blwr9\" (UniqueName: \"kubernetes.io/projected/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-kube-api-access-blwr9\") pod \"maas-api-revoke-sub-e5a70ecf-cmk4x\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:30.598954 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.598766 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a70ecf-cmk4x\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:30.598954 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.598788 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a70ecf-cmk4x\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:30.599470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.599438 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a70ecf-cmk4x\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:30.601194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.601172 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a70ecf-cmk4x\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:30.658022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.657952 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blwr9\" (UniqueName: \"kubernetes.io/projected/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-kube-api-access-blwr9\") pod \"maas-api-revoke-sub-e5a70ecf-cmk4x\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:30.697703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.697667 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:30.978934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.978899 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7"] Nov 03 22:23:30.984792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:30.984768 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:31.033996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.033969 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7"] Nov 03 22:23:31.080520 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:23:31.080490 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b049bc1_3d4a_4d78_ab23_b53dd95361d2.slice/crio-37f91563645b7f1fea5211d2270f51c1a1427090fbbe3d841bdc4f9a594cc8e3 WatchSource:0}: Error finding container 37f91563645b7f1fea5211d2270f51c1a1427090fbbe3d841bdc4f9a594cc8e3: Status 404 returned error can't find the container with id 37f91563645b7f1fea5211d2270f51c1a1427090fbbe3d841bdc4f9a594cc8e3 Nov 03 22:23:31.087239 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.087215 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x"] Nov 03 22:23:31.103962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.103939 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e715dfd3-3388-4668-8795-699bb857671d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2386f0-qzfj7\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:31.104067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.103980 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e715dfd3-3388-4668-8795-699bb857671d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2386f0-qzfj7\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:31.104153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.104128 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqwmp\" (UniqueName: \"kubernetes.io/projected/e715dfd3-3388-4668-8795-699bb857671d-kube-api-access-nqwmp\") pod \"maas-api-revoke-sub-aa2386f0-qzfj7\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:31.205742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.205710 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e715dfd3-3388-4668-8795-699bb857671d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2386f0-qzfj7\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:31.205908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.205755 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e715dfd3-3388-4668-8795-699bb857671d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2386f0-qzfj7\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:31.205908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.205786 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqwmp\" (UniqueName: \"kubernetes.io/projected/e715dfd3-3388-4668-8795-699bb857671d-kube-api-access-nqwmp\") pod \"maas-api-revoke-sub-aa2386f0-qzfj7\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:31.206524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.206494 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e715dfd3-3388-4668-8795-699bb857671d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2386f0-qzfj7\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:31.208045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.208022 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e715dfd3-3388-4668-8795-699bb857671d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2386f0-qzfj7\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:31.263607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.263578 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqwmp\" (UniqueName: \"kubernetes.io/projected/e715dfd3-3388-4668-8795-699bb857671d-kube-api-access-nqwmp\") pod \"maas-api-revoke-sub-aa2386f0-qzfj7\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:31.300081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.300057 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:31.666177 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:23:31.666146 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode715dfd3_3388_4668_8795_699bb857671d.slice/crio-3db555cd241a3085613e89218ee5042e795dd5d2be69b561ef8d86de74f2e235 WatchSource:0}: Error finding container 3db555cd241a3085613e89218ee5042e795dd5d2be69b561ef8d86de74f2e235: Status 404 returned error can't find the container with id 3db555cd241a3085613e89218ee5042e795dd5d2be69b561ef8d86de74f2e235 Nov 03 22:23:31.669854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.669833 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7"] Nov 03 22:23:31.879585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.879550 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" event={"ID":"e715dfd3-3388-4668-8795-699bb857671d","Type":"ContainerStarted","Data":"3db555cd241a3085613e89218ee5042e795dd5d2be69b561ef8d86de74f2e235"} Nov 03 22:23:31.881009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.880983 4298 generic.go:356] "Generic (PLEG): container finished" podID="1b049bc1-3d4a-4d78-ab23-b53dd95361d2" containerID="dc2769ccf4577a97ce18c48711ef6cfac792ced5846bce5b69748189433caef5" exitCode=0 Nov 03 22:23:31.881118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.881016 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" event={"ID":"1b049bc1-3d4a-4d78-ab23-b53dd95361d2","Type":"ContainerDied","Data":"dc2769ccf4577a97ce18c48711ef6cfac792ced5846bce5b69748189433caef5"} Nov 03 22:23:31.881118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:31.881052 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" event={"ID":"1b049bc1-3d4a-4d78-ab23-b53dd95361d2","Type":"ContainerStarted","Data":"37f91563645b7f1fea5211d2270f51c1a1427090fbbe3d841bdc4f9a594cc8e3"} Nov 03 22:23:32.888260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:32.888182 4298 generic.go:356] "Generic (PLEG): container finished" podID="e715dfd3-3388-4668-8795-699bb857671d" containerID="ed3091bc095b580ef16c06935475f047c0a28f7db07abe39b7d723fbc1d7d9a1" exitCode=0 Nov 03 22:23:32.888260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:32.888247 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" event={"ID":"e715dfd3-3388-4668-8795-699bb857671d","Type":"ContainerDied","Data":"ed3091bc095b580ef16c06935475f047c0a28f7db07abe39b7d723fbc1d7d9a1"} Nov 03 22:23:33.039615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.039591 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:33.125540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.125508 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-openshift-ca\") pod \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " Nov 03 22:23:33.125740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.125605 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-kube-api-access-blwr9\" (UniqueName: \"kubernetes.io/projected/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-kube-api-access-blwr9\") pod \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " Nov 03 22:23:33.125740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.125713 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-xks-ca\") pod \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\" (UID: \"1b049bc1-3d4a-4d78-ab23-b53dd95361d2\") " Nov 03 22:23:33.125920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.125890 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-openshift-ca" pod "1b049bc1-3d4a-4d78-ab23-b53dd95361d2" (UID: "1b049bc1-3d4a-4d78-ab23-b53dd95361d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:33.126034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.126018 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:33.127758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.127734 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-xks-ca" pod "1b049bc1-3d4a-4d78-ab23-b53dd95361d2" (UID: "1b049bc1-3d4a-4d78-ab23-b53dd95361d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:33.127851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.127814 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-kube-api-access-blwr9" pod "1b049bc1-3d4a-4d78-ab23-b53dd95361d2" (UID: "1b049bc1-3d4a-4d78-ab23-b53dd95361d2"). InnerVolumeSpecName "kube-api-access-blwr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:33.227337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.227299 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blwr9\" (UniqueName: \"kubernetes.io/projected/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-kube-api-access-blwr9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:33.227337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.227329 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b049bc1-3d4a-4d78-ab23-b53dd95361d2-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:33.278884 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.278854 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:33.894989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.894961 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" Nov 03 22:23:33.895453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.895000 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x" event={"ID":"1b049bc1-3d4a-4d78-ab23-b53dd95361d2","Type":"ContainerDied","Data":"37f91563645b7f1fea5211d2270f51c1a1427090fbbe3d841bdc4f9a594cc8e3"} Nov 03 22:23:33.895453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:33.895043 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f91563645b7f1fea5211d2270f51c1a1427090fbbe3d841bdc4f9a594cc8e3" Nov 03 22:23:34.037347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.037318 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:34.136952 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.136925 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e715dfd3-3388-4668-8795-699bb857671d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e715dfd3-3388-4668-8795-699bb857671d-maas-api-xks-ca\") pod \"e715dfd3-3388-4668-8795-699bb857671d\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " Nov 03 22:23:34.137099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.137074 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e715dfd3-3388-4668-8795-699bb857671d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e715dfd3-3388-4668-8795-699bb857671d-maas-api-openshift-ca\") pod \"e715dfd3-3388-4668-8795-699bb857671d\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " Nov 03 22:23:34.137152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.137123 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e715dfd3-3388-4668-8795-699bb857671d-kube-api-access-nqwmp\" (UniqueName: \"kubernetes.io/projected/e715dfd3-3388-4668-8795-699bb857671d-kube-api-access-nqwmp\") pod \"e715dfd3-3388-4668-8795-699bb857671d\" (UID: \"e715dfd3-3388-4668-8795-699bb857671d\") " Nov 03 22:23:34.137478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.137445 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e715dfd3-3388-4668-8795-699bb857671d-maas-api-openshift-ca" pod "e715dfd3-3388-4668-8795-699bb857671d" (UID: "e715dfd3-3388-4668-8795-699bb857671d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:34.139079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.139050 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e715dfd3-3388-4668-8795-699bb857671d-maas-api-xks-ca" pod "e715dfd3-3388-4668-8795-699bb857671d" (UID: "e715dfd3-3388-4668-8795-699bb857671d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:34.139486 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.139466 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e715dfd3-3388-4668-8795-699bb857671d-kube-api-access-nqwmp" pod "e715dfd3-3388-4668-8795-699bb857671d" (UID: "e715dfd3-3388-4668-8795-699bb857671d"). InnerVolumeSpecName "kube-api-access-nqwmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:34.238700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.238666 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e715dfd3-3388-4668-8795-699bb857671d-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:34.238700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.238696 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e715dfd3-3388-4668-8795-699bb857671d-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:34.238700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.238706 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqwmp\" (UniqueName: \"kubernetes.io/projected/e715dfd3-3388-4668-8795-699bb857671d-kube-api-access-nqwmp\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:34.901465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.901430 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" event={"ID":"e715dfd3-3388-4668-8795-699bb857671d","Type":"ContainerDied","Data":"3db555cd241a3085613e89218ee5042e795dd5d2be69b561ef8d86de74f2e235"} Nov 03 22:23:34.901465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.901450 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7" Nov 03 22:23:34.901939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:34.901479 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3db555cd241a3085613e89218ee5042e795dd5d2be69b561ef8d86de74f2e235" Nov 03 22:23:37.790579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:37.790543 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:38.891502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:38.891470 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:41.631677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.631629 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm"] Nov 03 22:23:41.632172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.632157 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1b049bc1-3d4a-4d78-ab23-b53dd95361d2" containerName="revoke-keys" Nov 03 22:23:41.632214 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.632173 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b049bc1-3d4a-4d78-ab23-b53dd95361d2" containerName="revoke-keys" Nov 03 22:23:41.632214 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.632195 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e715dfd3-3388-4668-8795-699bb857671d" containerName="revoke-keys" Nov 03 22:23:41.632214 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.632200 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e715dfd3-3388-4668-8795-699bb857671d" containerName="revoke-keys" Nov 03 22:23:41.632317 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.632274 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b049bc1-3d4a-4d78-ab23-b53dd95361d2" containerName="revoke-keys" Nov 03 22:23:41.632317 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.632285 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e715dfd3-3388-4668-8795-699bb857671d" containerName="revoke-keys" Nov 03 22:23:41.639729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.637947 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:41.687258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.687223 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm"] Nov 03 22:23:41.735536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.735509 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:23:41.735725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.735551 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:23:41.820741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.820706 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqcxg\" (UniqueName: \"kubernetes.io/projected/021f11bd-d598-4905-b38f-55bb296889e8-kube-api-access-kqcxg\") pod \"maas-api-revoke-sub-0fa1d926-hffqm\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:41.820906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.820746 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fa1d926-hffqm\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:41.820906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.820776 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fa1d926-hffqm\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:41.922208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.922123 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fa1d926-hffqm\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:41.922360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.922285 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqcxg\" (UniqueName: \"kubernetes.io/projected/021f11bd-d598-4905-b38f-55bb296889e8-kube-api-access-kqcxg\") pod \"maas-api-revoke-sub-0fa1d926-hffqm\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:41.922360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.922311 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fa1d926-hffqm\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:41.922995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.922976 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fa1d926-hffqm\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:41.924536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.924516 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fa1d926-hffqm\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:41.980144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:41.980111 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqcxg\" (UniqueName: \"kubernetes.io/projected/021f11bd-d598-4905-b38f-55bb296889e8-kube-api-access-kqcxg\") pod \"maas-api-revoke-sub-0fa1d926-hffqm\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:42.253041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:42.253004 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:42.376680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:42.376631 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:42.625243 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:23:42.625211 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021f11bd_d598_4905_b38f_55bb296889e8.slice/crio-46ddbb45523f5c4fb6f1019310fc7b17f20d7609ee3af9807d0a2b85fe65c9b6 WatchSource:0}: Error finding container 46ddbb45523f5c4fb6f1019310fc7b17f20d7609ee3af9807d0a2b85fe65c9b6: Status 404 returned error can't find the container with id 46ddbb45523f5c4fb6f1019310fc7b17f20d7609ee3af9807d0a2b85fe65c9b6 Nov 03 22:23:42.629777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:42.629748 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm"] Nov 03 22:23:42.949357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:42.949331 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" event={"ID":"021f11bd-d598-4905-b38f-55bb296889e8","Type":"ContainerStarted","Data":"185ed392240caa96d2c83bfed586da7f9ddfdcb9bfe371398b8c14af77da8ebd"} Nov 03 22:23:42.949700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:42.949381 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" event={"ID":"021f11bd-d598-4905-b38f-55bb296889e8","Type":"ContainerStarted","Data":"46ddbb45523f5c4fb6f1019310fc7b17f20d7609ee3af9807d0a2b85fe65c9b6"} Nov 03 22:23:43.067469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:43.067416 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" podStartSLOduration=2.067401868 podStartE2EDuration="2.067401868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:43.065298331 +0000 UTC m=+980.046031984" watchObservedRunningTime="2026-11-03 22:23:43.067401868 +0000 UTC m=+980.048135520" Nov 03 22:23:43.956914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:43.956878 4298 generic.go:356] "Generic (PLEG): container finished" podID="021f11bd-d598-4905-b38f-55bb296889e8" containerID="185ed392240caa96d2c83bfed586da7f9ddfdcb9bfe371398b8c14af77da8ebd" exitCode=0 Nov 03 22:23:43.957348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:43.956966 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" event={"ID":"021f11bd-d598-4905-b38f-55bb296889e8","Type":"ContainerDied","Data":"185ed392240caa96d2c83bfed586da7f9ddfdcb9bfe371398b8c14af77da8ebd"} Nov 03 22:23:45.193330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.193302 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:45.361536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.361447 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-xks-ca\") pod \"021f11bd-d598-4905-b38f-55bb296889e8\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " Nov 03 22:23:45.361706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.361628 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021f11bd-d598-4905-b38f-55bb296889e8-kube-api-access-kqcxg\" (UniqueName: \"kubernetes.io/projected/021f11bd-d598-4905-b38f-55bb296889e8-kube-api-access-kqcxg\") pod \"021f11bd-d598-4905-b38f-55bb296889e8\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " Nov 03 22:23:45.361772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.361727 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-openshift-ca\") pod \"021f11bd-d598-4905-b38f-55bb296889e8\" (UID: \"021f11bd-d598-4905-b38f-55bb296889e8\") " Nov 03 22:23:45.362213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.362182 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-openshift-ca" pod "021f11bd-d598-4905-b38f-55bb296889e8" (UID: "021f11bd-d598-4905-b38f-55bb296889e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:45.363746 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.363706 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-xks-ca" pod "021f11bd-d598-4905-b38f-55bb296889e8" (UID: "021f11bd-d598-4905-b38f-55bb296889e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:45.363746 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.363722 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021f11bd-d598-4905-b38f-55bb296889e8-kube-api-access-kqcxg" pod "021f11bd-d598-4905-b38f-55bb296889e8" (UID: "021f11bd-d598-4905-b38f-55bb296889e8"). InnerVolumeSpecName "kube-api-access-kqcxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:45.463804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.463755 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqcxg\" (UniqueName: \"kubernetes.io/projected/021f11bd-d598-4905-b38f-55bb296889e8-kube-api-access-kqcxg\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:45.463804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.463799 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:45.464065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.463818 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021f11bd-d598-4905-b38f-55bb296889e8-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:45.978957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.978926 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" Nov 03 22:23:45.978957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.978938 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm" event={"ID":"021f11bd-d598-4905-b38f-55bb296889e8","Type":"ContainerDied","Data":"46ddbb45523f5c4fb6f1019310fc7b17f20d7609ee3af9807d0a2b85fe65c9b6"} Nov 03 22:23:45.979204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:45.979016 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46ddbb45523f5c4fb6f1019310fc7b17f20d7609ee3af9807d0a2b85fe65c9b6" Nov 03 22:23:46.648220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.648185 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld"] Nov 03 22:23:46.648738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.648723 4298 cpu_manager.go:414] "containerMap: removing container" podUID="021f11bd-d598-4905-b38f-55bb296889e8" containerName="revoke-keys" Nov 03 22:23:46.648787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.648739 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021f11bd-d598-4905-b38f-55bb296889e8" containerName="revoke-keys" Nov 03 22:23:46.648838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.648828 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="021f11bd-d598-4905-b38f-55bb296889e8" containerName="revoke-keys" Nov 03 22:23:46.652310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.652288 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:46.707024 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.706993 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld"] Nov 03 22:23:46.754968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.754935 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:23:46.754968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.754941 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:23:46.779157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.779123 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzmgx\" (UniqueName: \"kubernetes.io/projected/397ed36d-1d6d-47da-bde1-fca932d0951a-kube-api-access-wzmgx\") pod \"maas-api-revoke-sub-1adc586f-4hhld\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:46.779348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.779248 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1adc586f-4hhld\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:46.779405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.779377 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1adc586f-4hhld\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:46.881065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.881030 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzmgx\" (UniqueName: \"kubernetes.io/projected/397ed36d-1d6d-47da-bde1-fca932d0951a-kube-api-access-wzmgx\") pod \"maas-api-revoke-sub-1adc586f-4hhld\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:46.881253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.881098 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1adc586f-4hhld\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:46.881253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.881194 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1adc586f-4hhld\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:46.881783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.881756 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1adc586f-4hhld\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:46.883449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.883428 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1adc586f-4hhld\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:46.939540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.939474 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzmgx\" (UniqueName: \"kubernetes.io/projected/397ed36d-1d6d-47da-bde1-fca932d0951a-kube-api-access-wzmgx\") pod \"maas-api-revoke-sub-1adc586f-4hhld\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:46.966222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:46.966192 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:47.007879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.007841 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk"] Nov 03 22:23:47.012574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.012549 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:47.075745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.075713 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk"] Nov 03 22:23:47.184985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.184946 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b9c655-b581-4633-9030-4a5590464160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39feb93e-44pqk\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:47.185144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.184994 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhd7b\" (UniqueName: \"kubernetes.io/projected/d0b9c655-b581-4633-9030-4a5590464160-kube-api-access-dhd7b\") pod \"maas-api-revoke-sub-39feb93e-44pqk\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:47.185202 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.185179 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b9c655-b581-4633-9030-4a5590464160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39feb93e-44pqk\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:47.206006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.205976 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:47.286234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.286195 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b9c655-b581-4633-9030-4a5590464160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39feb93e-44pqk\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:47.286234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.286232 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhd7b\" (UniqueName: \"kubernetes.io/projected/d0b9c655-b581-4633-9030-4a5590464160-kube-api-access-dhd7b\") pod \"maas-api-revoke-sub-39feb93e-44pqk\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:47.286484 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.286327 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b9c655-b581-4633-9030-4a5590464160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39feb93e-44pqk\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:47.286995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.286974 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b9c655-b581-4633-9030-4a5590464160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39feb93e-44pqk\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:47.288543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.288521 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b9c655-b581-4633-9030-4a5590464160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39feb93e-44pqk\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:47.341050 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:23:47.341026 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397ed36d_1d6d_47da_bde1_fca932d0951a.slice/crio-7b0d191f82e4cc024189dc4fd7865f8fcaa7bb830aa555b423a751cd408216d0 WatchSource:0}: Error finding container 7b0d191f82e4cc024189dc4fd7865f8fcaa7bb830aa555b423a751cd408216d0: Status 404 returned error can't find the container with id 7b0d191f82e4cc024189dc4fd7865f8fcaa7bb830aa555b423a751cd408216d0 Nov 03 22:23:47.343995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.343972 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld"] Nov 03 22:23:47.345905 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.345883 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhd7b\" (UniqueName: \"kubernetes.io/projected/d0b9c655-b581-4633-9030-4a5590464160-kube-api-access-dhd7b\") pod \"maas-api-revoke-sub-39feb93e-44pqk\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:47.553122 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:23:47.553082 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:23:47.637812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.637787 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:47.993003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.992970 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" event={"ID":"397ed36d-1d6d-47da-bde1-fca932d0951a","Type":"ContainerStarted","Data":"2e5a07ec6c40f9c8942cb8f8f66022e0fcfe4ebe55f8e651569dbabaaf351ca8"} Nov 03 22:23:47.993406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:47.993027 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" event={"ID":"397ed36d-1d6d-47da-bde1-fca932d0951a","Type":"ContainerStarted","Data":"7b0d191f82e4cc024189dc4fd7865f8fcaa7bb830aa555b423a751cd408216d0"} Nov 03 22:23:48.004987 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:23:48.004957 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b9c655_b581_4633_9030_4a5590464160.slice/crio-ee862c148d8e9b15b91c975c0bc4ebb9951d488c2cfe884cadd2f72fc3526f9c WatchSource:0}: Error finding container ee862c148d8e9b15b91c975c0bc4ebb9951d488c2cfe884cadd2f72fc3526f9c: Status 404 returned error can't find the container with id ee862c148d8e9b15b91c975c0bc4ebb9951d488c2cfe884cadd2f72fc3526f9c Nov 03 22:23:48.008219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.008198 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk"] Nov 03 22:23:48.108686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.108615 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" podStartSLOduration=2.108601444 podStartE2EDuration="2.108601444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:48.107582572 +0000 UTC m=+985.088316216" watchObservedRunningTime="2026-11-03 22:23:48.108601444 +0000 UTC m=+985.089335097" Nov 03 22:23:48.355994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.355964 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5"] Nov 03 22:23:48.359738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.359717 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.399612 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.399576 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.399787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.399621 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.399787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.399732 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.399787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.399763 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktlfm\" (UniqueName: \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-kube-api-access-ktlfm\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.399787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.399780 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.399947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.399928 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.399983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.399960 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.399983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.399976 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.400065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.400046 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.413367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.413331 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3378fd-openshift-default-dockercfg-7fv79\"" Nov 03 22:23:48.415800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.415775 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5"] Nov 03 22:23:48.501346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501266 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.501346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501319 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktlfm\" (UniqueName: \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-kube-api-access-ktlfm\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.501556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501347 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.501556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501439 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.501556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501474 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.501556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501521 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.501814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501575 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.501814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501681 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.501814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501738 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.501814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501761 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.502033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501859 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.502033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.501951 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.502033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.502007 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.502345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.502321 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.504021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.504000 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.504135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.504116 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.559326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.559299 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.559478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.559367 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktlfm\" (UniqueName: \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-kube-api-access-ktlfm\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.672864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.672829 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:48.999677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.999626 4298 generic.go:356] "Generic (PLEG): container finished" podID="397ed36d-1d6d-47da-bde1-fca932d0951a" containerID="2e5a07ec6c40f9c8942cb8f8f66022e0fcfe4ebe55f8e651569dbabaaf351ca8" exitCode=0 Nov 03 22:23:49.000148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:48.999709 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" event={"ID":"397ed36d-1d6d-47da-bde1-fca932d0951a","Type":"ContainerDied","Data":"2e5a07ec6c40f9c8942cb8f8f66022e0fcfe4ebe55f8e651569dbabaaf351ca8"} Nov 03 22:23:49.001286 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:49.001263 4298 generic.go:356] "Generic (PLEG): container finished" podID="d0b9c655-b581-4633-9030-4a5590464160" containerID="c1990bd8146a02595bb8bf78c0f5b4f9212e382b233b99b6bc606da959991bbe" exitCode=0 Nov 03 22:23:49.001406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:49.001333 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" event={"ID":"d0b9c655-b581-4633-9030-4a5590464160","Type":"ContainerDied","Data":"c1990bd8146a02595bb8bf78c0f5b4f9212e382b233b99b6bc606da959991bbe"} Nov 03 22:23:49.001406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:49.001371 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" event={"ID":"d0b9c655-b581-4633-9030-4a5590464160","Type":"ContainerStarted","Data":"ee862c148d8e9b15b91c975c0bc4ebb9951d488c2cfe884cadd2f72fc3526f9c"} Nov 03 22:23:49.092235 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:23:49.092200 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c2e0c5_cb35_4ce7_a07a_bae61ef1f851.slice/crio-17e9681f4410eee754fa0a8f190f114feb31a60ac76ca04b619f28dad8990447 WatchSource:0}: Error finding container 17e9681f4410eee754fa0a8f190f114feb31a60ac76ca04b619f28dad8990447: Status 404 returned error can't find the container with id 17e9681f4410eee754fa0a8f190f114feb31a60ac76ca04b619f28dad8990447 Nov 03 22:23:49.094906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:49.094872 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:23:49.095022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:49.094937 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:23:49.095022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:49.094965 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:23:49.095867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:49.095843 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5"] Nov 03 22:23:50.008919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.008877 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" event={"ID":"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851","Type":"ContainerStarted","Data":"bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30"} Nov 03 22:23:50.009388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.008938 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" event={"ID":"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851","Type":"ContainerStarted","Data":"17e9681f4410eee754fa0a8f190f114feb31a60ac76ca04b619f28dad8990447"} Nov 03 22:23:50.249498 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.249221 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:50.252411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.252384 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:50.350666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.350547 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0b9c655-b581-4633-9030-4a5590464160-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b9c655-b581-4633-9030-4a5590464160-maas-api-openshift-ca\") pod \"d0b9c655-b581-4633-9030-4a5590464160\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " Nov 03 22:23:50.350837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.350706 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b9c655-b581-4633-9030-4a5590464160-kube-api-access-dhd7b\" (UniqueName: \"kubernetes.io/projected/d0b9c655-b581-4633-9030-4a5590464160-kube-api-access-dhd7b\") pod \"d0b9c655-b581-4633-9030-4a5590464160\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " Nov 03 22:23:50.350837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.350756 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0b9c655-b581-4633-9030-4a5590464160-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b9c655-b581-4633-9030-4a5590464160-maas-api-xks-ca\") pod \"d0b9c655-b581-4633-9030-4a5590464160\" (UID: \"d0b9c655-b581-4633-9030-4a5590464160\") " Nov 03 22:23:50.350837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.350831 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/397ed36d-1d6d-47da-bde1-fca932d0951a-kube-api-access-wzmgx\" (UniqueName: \"kubernetes.io/projected/397ed36d-1d6d-47da-bde1-fca932d0951a-kube-api-access-wzmgx\") pod \"397ed36d-1d6d-47da-bde1-fca932d0951a\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " Nov 03 22:23:50.351035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.350881 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-xks-ca\") pod \"397ed36d-1d6d-47da-bde1-fca932d0951a\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " Nov 03 22:23:50.351035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.350931 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-openshift-ca\") pod \"397ed36d-1d6d-47da-bde1-fca932d0951a\" (UID: \"397ed36d-1d6d-47da-bde1-fca932d0951a\") " Nov 03 22:23:50.351144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.351021 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b9c655-b581-4633-9030-4a5590464160-maas-api-openshift-ca" pod "d0b9c655-b581-4633-9030-4a5590464160" (UID: "d0b9c655-b581-4633-9030-4a5590464160"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:50.351598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.351364 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b9c655-b581-4633-9030-4a5590464160-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.351866 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.351805 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-openshift-ca" pod "397ed36d-1d6d-47da-bde1-fca932d0951a" (UID: "397ed36d-1d6d-47da-bde1-fca932d0951a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:50.353558 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.353521 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0b9c655-b581-4633-9030-4a5590464160-maas-api-xks-ca" pod "d0b9c655-b581-4633-9030-4a5590464160" (UID: "d0b9c655-b581-4633-9030-4a5590464160"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:50.353728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.353709 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397ed36d-1d6d-47da-bde1-fca932d0951a-kube-api-access-wzmgx" pod "397ed36d-1d6d-47da-bde1-fca932d0951a" (UID: "397ed36d-1d6d-47da-bde1-fca932d0951a"). InnerVolumeSpecName "kube-api-access-wzmgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:50.354191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.354171 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-xks-ca" pod "397ed36d-1d6d-47da-bde1-fca932d0951a" (UID: "397ed36d-1d6d-47da-bde1-fca932d0951a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:50.354282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.354264 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b9c655-b581-4633-9030-4a5590464160-kube-api-access-dhd7b" pod "d0b9c655-b581-4633-9030-4a5590464160" (UID: "d0b9c655-b581-4633-9030-4a5590464160"). InnerVolumeSpecName "kube-api-access-dhd7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:50.366515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.366473 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" podStartSLOduration=2.366457955 podStartE2EDuration="2.366457955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:50.157020984 +0000 UTC m=+987.137754638" watchObservedRunningTime="2026-11-03 22:23:50.366457955 +0000 UTC m=+987.347191608" Nov 03 22:23:50.452776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.452742 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhd7b\" (UniqueName: \"kubernetes.io/projected/d0b9c655-b581-4633-9030-4a5590464160-kube-api-access-dhd7b\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.452776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.452774 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b9c655-b581-4633-9030-4a5590464160-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.452987 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.452787 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzmgx\" (UniqueName: \"kubernetes.io/projected/397ed36d-1d6d-47da-bde1-fca932d0951a-kube-api-access-wzmgx\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.452987 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.452799 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.452987 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.452810 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397ed36d-1d6d-47da-bde1-fca932d0951a-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.674021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.673930 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:50.675750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.675725 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:23:50.675881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:50.675778 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:23:51.016127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:51.016086 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" event={"ID":"397ed36d-1d6d-47da-bde1-fca932d0951a","Type":"ContainerDied","Data":"7b0d191f82e4cc024189dc4fd7865f8fcaa7bb830aa555b423a751cd408216d0"} Nov 03 22:23:51.016127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:51.016112 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld" Nov 03 22:23:51.016705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:51.016154 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b0d191f82e4cc024189dc4fd7865f8fcaa7bb830aa555b423a751cd408216d0" Nov 03 22:23:51.017430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:51.017409 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" Nov 03 22:23:51.017430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:51.017427 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk" event={"ID":"d0b9c655-b581-4633-9030-4a5590464160","Type":"ContainerDied","Data":"ee862c148d8e9b15b91c975c0bc4ebb9951d488c2cfe884cadd2f72fc3526f9c"} Nov 03 22:23:51.017638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:51.017474 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee862c148d8e9b15b91c975c0bc4ebb9951d488c2cfe884cadd2f72fc3526f9c" Nov 03 22:23:51.566937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:51.566907 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:51.674199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:51.674157 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:23:51.674369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:51.674215 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:23:52.673879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:52.673848 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:23:52.674282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:52.673910 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:23:53.673172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:53.673134 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:23:53.673359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:53.673194 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:23:54.673962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:54.673927 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:23:54.674358 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:54.673988 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:23:54.915923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:54.915890 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg"] Nov 03 22:23:54.916376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:54.916358 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d0b9c655-b581-4633-9030-4a5590464160" containerName="revoke-keys" Nov 03 22:23:54.916376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:54.916374 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b9c655-b581-4633-9030-4a5590464160" containerName="revoke-keys" Nov 03 22:23:54.916580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:54.916407 4298 cpu_manager.go:414] "containerMap: removing container" podUID="397ed36d-1d6d-47da-bde1-fca932d0951a" containerName="revoke-keys" Nov 03 22:23:54.916580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:54.916412 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397ed36d-1d6d-47da-bde1-fca932d0951a" containerName="revoke-keys" Nov 03 22:23:54.916580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:54.916501 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="397ed36d-1d6d-47da-bde1-fca932d0951a" containerName="revoke-keys" Nov 03 22:23:54.916580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:54.916510 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b9c655-b581-4633-9030-4a5590464160" containerName="revoke-keys" Nov 03 22:23:54.920975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:54.920951 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:54.973353 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:54.973317 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg"] Nov 03 22:23:55.003600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.003560 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-890b359d-lrtzg\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:55.003600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.003603 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-890b359d-lrtzg\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:55.003823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.003732 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qt8m\" (UniqueName: \"kubernetes.io/projected/a78a206a-9a9d-4286-a4a4-b12929c75977-kube-api-access-4qt8m\") pod \"maas-api-revoke-sub-890b359d-lrtzg\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:55.020383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.020356 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:23:55.020383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.020376 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:23:55.104986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.104951 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qt8m\" (UniqueName: \"kubernetes.io/projected/a78a206a-9a9d-4286-a4a4-b12929c75977-kube-api-access-4qt8m\") pod \"maas-api-revoke-sub-890b359d-lrtzg\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:55.105183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.105061 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-890b359d-lrtzg\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:55.105183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.105100 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-890b359d-lrtzg\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:55.105709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.105688 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-890b359d-lrtzg\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:55.107464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.107440 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-890b359d-lrtzg\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:55.163406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.163371 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qt8m\" (UniqueName: \"kubernetes.io/projected/a78a206a-9a9d-4286-a4a4-b12929c75977-kube-api-access-4qt8m\") pod \"maas-api-revoke-sub-890b359d-lrtzg\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:55.260802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.260724 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:55.626766 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:23:55.626734 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78a206a_9a9d_4286_a4a4_b12929c75977.slice/crio-51c960730e50b788ba4c3e7d2922e71a1630737d7510b8cae95a6e6739efb4fc WatchSource:0}: Error finding container 51c960730e50b788ba4c3e7d2922e71a1630737d7510b8cae95a6e6739efb4fc: Status 404 returned error can't find the container with id 51c960730e50b788ba4c3e7d2922e71a1630737d7510b8cae95a6e6739efb4fc Nov 03 22:23:55.629870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.629850 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg"] Nov 03 22:23:55.673804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.673770 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:23:55.673934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:55.673835 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:23:56.074991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:56.074952 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" event={"ID":"a78a206a-9a9d-4286-a4a4-b12929c75977","Type":"ContainerStarted","Data":"284389027edc742a1e81518d6a7b6dc7645d0f9801a9b6a113fdd5f5ea5f483f"} Nov 03 22:23:56.075385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:56.075012 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" event={"ID":"a78a206a-9a9d-4286-a4a4-b12929c75977","Type":"ContainerStarted","Data":"51c960730e50b788ba4c3e7d2922e71a1630737d7510b8cae95a6e6739efb4fc"} Nov 03 22:23:56.081298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:56.081264 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:56.192454 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:56.192336 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" podStartSLOduration=2.19231466 podStartE2EDuration="2.19231466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:56.190268281 +0000 UTC m=+993.171001934" watchObservedRunningTime="2026-11-03 22:23:56.19231466 +0000 UTC m=+993.173048316" Nov 03 22:23:56.678124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:56.678095 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:56.678517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:56.678496 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:56.679227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:56.679209 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:23:57.082560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:57.082529 4298 generic.go:356] "Generic (PLEG): container finished" podID="a78a206a-9a9d-4286-a4a4-b12929c75977" containerID="284389027edc742a1e81518d6a7b6dc7645d0f9801a9b6a113fdd5f5ea5f483f" exitCode=0 Nov 03 22:23:57.082986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:57.082609 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" event={"ID":"a78a206a-9a9d-4286-a4a4-b12929c75977","Type":"ContainerDied","Data":"284389027edc742a1e81518d6a7b6dc7645d0f9801a9b6a113fdd5f5ea5f483f"} Nov 03 22:23:58.246096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:58.246061 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:58.364479 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:58.364402 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78a206a-9a9d-4286-a4a4-b12929c75977-kube-api-access-4qt8m\" (UniqueName: \"kubernetes.io/projected/a78a206a-9a9d-4286-a4a4-b12929c75977-kube-api-access-4qt8m\") pod \"a78a206a-9a9d-4286-a4a4-b12929c75977\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " Nov 03 22:23:58.364479 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:58.364467 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-openshift-ca\") pod \"a78a206a-9a9d-4286-a4a4-b12929c75977\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " Nov 03 22:23:58.364719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:58.364586 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-xks-ca\") pod \"a78a206a-9a9d-4286-a4a4-b12929c75977\" (UID: \"a78a206a-9a9d-4286-a4a4-b12929c75977\") " Nov 03 22:23:58.364920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:58.364890 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-openshift-ca" pod "a78a206a-9a9d-4286-a4a4-b12929c75977" (UID: "a78a206a-9a9d-4286-a4a4-b12929c75977"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:58.366698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:58.366666 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-xks-ca" pod "a78a206a-9a9d-4286-a4a4-b12929c75977" (UID: "a78a206a-9a9d-4286-a4a4-b12929c75977"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:58.367064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:58.367042 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78a206a-9a9d-4286-a4a4-b12929c75977-kube-api-access-4qt8m" pod "a78a206a-9a9d-4286-a4a4-b12929c75977" (UID: "a78a206a-9a9d-4286-a4a4-b12929c75977"). InnerVolumeSpecName "kube-api-access-4qt8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:58.466384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:58.466348 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:58.466384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:58.466378 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a78a206a-9a9d-4286-a4a4-b12929c75977-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:58.466384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:58.466388 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qt8m\" (UniqueName: \"kubernetes.io/projected/a78a206a-9a9d-4286-a4a4-b12929c75977-kube-api-access-4qt8m\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:59.096273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:59.096245 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" Nov 03 22:23:59.096453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:59.096277 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg" event={"ID":"a78a206a-9a9d-4286-a4a4-b12929c75977","Type":"ContainerDied","Data":"51c960730e50b788ba4c3e7d2922e71a1630737d7510b8cae95a6e6739efb4fc"} Nov 03 22:23:59.096453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:23:59.096332 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51c960730e50b788ba4c3e7d2922e71a1630737d7510b8cae95a6e6739efb4fc" Nov 03 22:24:00.595154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:00.595119 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:02.105214 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.105179 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7"] Nov 03 22:24:02.105718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.105703 4298 cpu_manager.go:414] "containerMap: removing container" podUID="a78a206a-9a9d-4286-a4a4-b12929c75977" containerName="revoke-keys" Nov 03 22:24:02.105763 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.105718 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78a206a-9a9d-4286-a4a4-b12929c75977" containerName="revoke-keys" Nov 03 22:24:02.105795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.105787 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78a206a-9a9d-4286-a4a4-b12929c75977" containerName="revoke-keys" Nov 03 22:24:02.112440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.110624 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:02.162110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.162075 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7"] Nov 03 22:24:02.209608 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.209576 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:24:02.209795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.209635 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:24:02.211136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.211113 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f93848d8-bp9k7\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:02.211211 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.211189 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f93848d8-bp9k7\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:02.211292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.211279 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vh4\" (UniqueName: \"kubernetes.io/projected/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-kube-api-access-96vh4\") pod \"maas-api-revoke-sub-f93848d8-bp9k7\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:02.312192 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.312150 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vh4\" (UniqueName: \"kubernetes.io/projected/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-kube-api-access-96vh4\") pod \"maas-api-revoke-sub-f93848d8-bp9k7\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:02.312379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.312295 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f93848d8-bp9k7\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:02.312379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.312336 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f93848d8-bp9k7\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:02.312999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.312981 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f93848d8-bp9k7\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:02.314791 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.314771 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f93848d8-bp9k7\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:02.369800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.369723 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vh4\" (UniqueName: \"kubernetes.io/projected/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-kube-api-access-96vh4\") pod \"maas-api-revoke-sub-f93848d8-bp9k7\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:02.431806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.431756 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:02.799393 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:24:02.799354 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb954196_4b0b_4777_bb5b_ecb8fd5328a1.slice/crio-d83e4b6ebca40a4e8a68beeaa3f02a939c5a09be7016026cce3db79162ce29c7 WatchSource:0}: Error finding container d83e4b6ebca40a4e8a68beeaa3f02a939c5a09be7016026cce3db79162ce29c7: Status 404 returned error can't find the container with id d83e4b6ebca40a4e8a68beeaa3f02a939c5a09be7016026cce3db79162ce29c7 Nov 03 22:24:02.802530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:02.802506 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7"] Nov 03 22:24:03.127230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:03.127200 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" event={"ID":"cb954196-4b0b-4777-bb5b-ecb8fd5328a1","Type":"ContainerStarted","Data":"d83e4b6ebca40a4e8a68beeaa3f02a939c5a09be7016026cce3db79162ce29c7"} Nov 03 22:24:04.135885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:04.135788 4298 generic.go:356] "Generic (PLEG): container finished" podID="cb954196-4b0b-4777-bb5b-ecb8fd5328a1" containerID="aa05e5e4d18f12476680265fd74c35e88628cb6f75d706d0e1612eff196381c1" exitCode=0 Nov 03 22:24:04.136283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:04.135872 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" event={"ID":"cb954196-4b0b-4777-bb5b-ecb8fd5328a1","Type":"ContainerDied","Data":"aa05e5e4d18f12476680265fd74c35e88628cb6f75d706d0e1612eff196381c1"} Nov 03 22:24:05.085311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:05.085279 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:05.283339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:05.283313 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:05.448950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:05.448925 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-openshift-ca\") pod \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " Nov 03 22:24:05.449108 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:05.448972 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-kube-api-access-96vh4\" (UniqueName: \"kubernetes.io/projected/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-kube-api-access-96vh4\") pod \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " Nov 03 22:24:05.449108 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:05.449092 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-xks-ca\") pod \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\" (UID: \"cb954196-4b0b-4777-bb5b-ecb8fd5328a1\") " Nov 03 22:24:05.449312 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:05.449287 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-openshift-ca" pod "cb954196-4b0b-4777-bb5b-ecb8fd5328a1" (UID: "cb954196-4b0b-4777-bb5b-ecb8fd5328a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:24:05.449571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:05.449547 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:05.451133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:05.451108 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-xks-ca" pod "cb954196-4b0b-4777-bb5b-ecb8fd5328a1" (UID: "cb954196-4b0b-4777-bb5b-ecb8fd5328a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:05.451220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:05.451186 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-kube-api-access-96vh4" pod "cb954196-4b0b-4777-bb5b-ecb8fd5328a1" (UID: "cb954196-4b0b-4777-bb5b-ecb8fd5328a1"). InnerVolumeSpecName "kube-api-access-96vh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:24:05.551214 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:05.551183 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96vh4\" (UniqueName: \"kubernetes.io/projected/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-kube-api-access-96vh4\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:05.551214 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:05.551217 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb954196-4b0b-4777-bb5b-ecb8fd5328a1-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:06.149163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:06.149130 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" Nov 03 22:24:06.149317 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:06.149158 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7" event={"ID":"cb954196-4b0b-4777-bb5b-ecb8fd5328a1","Type":"ContainerDied","Data":"d83e4b6ebca40a4e8a68beeaa3f02a939c5a09be7016026cce3db79162ce29c7"} Nov 03 22:24:06.149317 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:06.149206 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d83e4b6ebca40a4e8a68beeaa3f02a939c5a09be7016026cce3db79162ce29c7" Nov 03 22:24:08.131557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:08.131520 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:08.310733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:08.310706 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:09.684120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:09.684090 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:10.671786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.671745 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9"] Nov 03 22:24:10.672259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.672246 4298 cpu_manager.go:414] "containerMap: removing container" podUID="cb954196-4b0b-4777-bb5b-ecb8fd5328a1" containerName="revoke-keys" Nov 03 22:24:10.672333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.672260 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb954196-4b0b-4777-bb5b-ecb8fd5328a1" containerName="revoke-keys" Nov 03 22:24:10.672392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.672382 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb954196-4b0b-4777-bb5b-ecb8fd5328a1" containerName="revoke-keys" Nov 03 22:24:10.675905 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.675882 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:10.727060 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.727014 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12fe3022-k8rh9\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:10.727590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.727081 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12fe3022-k8rh9\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:10.727590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.727173 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbpn4\" (UniqueName: \"kubernetes.io/projected/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-kube-api-access-cbpn4\") pod \"maas-api-revoke-sub-12fe3022-k8rh9\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:10.728870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.728840 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9"] Nov 03 22:24:10.775476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.775439 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:24:10.775707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.775510 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:24:10.827785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.827740 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12fe3022-k8rh9\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:10.827946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.827805 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12fe3022-k8rh9\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:10.827946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.827879 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbpn4\" (UniqueName: \"kubernetes.io/projected/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-kube-api-access-cbpn4\") pod \"maas-api-revoke-sub-12fe3022-k8rh9\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:10.828476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.828442 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12fe3022-k8rh9\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:10.830350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.830327 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12fe3022-k8rh9\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:10.885559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.885519 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbpn4\" (UniqueName: \"kubernetes.io/projected/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-kube-api-access-cbpn4\") pod \"maas-api-revoke-sub-12fe3022-k8rh9\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:10.988985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:10.988947 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:11.362990 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:24:11.362955 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355d6d35_1bf8_4e19_8eeb_ed6d6c026b65.slice/crio-f6f5ffb0d229229ede85047019361dbd4ffc842d50781683d49c8ec9286e62dd WatchSource:0}: Error finding container f6f5ffb0d229229ede85047019361dbd4ffc842d50781683d49c8ec9286e62dd: Status 404 returned error can't find the container with id f6f5ffb0d229229ede85047019361dbd4ffc842d50781683d49c8ec9286e62dd Nov 03 22:24:11.366668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:11.366622 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9"] Nov 03 22:24:12.217343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:12.217309 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" event={"ID":"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65","Type":"ContainerStarted","Data":"f07aa569e6d3b2b9876cf351304737e3ba3da63efd5e1ea62549fecd46570d7a"} Nov 03 22:24:12.217736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:12.217362 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" event={"ID":"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65","Type":"ContainerStarted","Data":"f6f5ffb0d229229ede85047019361dbd4ffc842d50781683d49c8ec9286e62dd"} Nov 03 22:24:12.334678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:12.334610 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" podStartSLOduration=2.334597497 podStartE2EDuration="2.334597497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:12.331836133 +0000 UTC m=+1009.312569786" watchObservedRunningTime="2026-11-03 22:24:12.334597497 +0000 UTC m=+1009.315331149" Nov 03 22:24:13.224327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:13.224285 4298 generic.go:356] "Generic (PLEG): container finished" podID="355d6d35-1bf8-4e19-8eeb-ed6d6c026b65" containerID="f07aa569e6d3b2b9876cf351304737e3ba3da63efd5e1ea62549fecd46570d7a" exitCode=0 Nov 03 22:24:13.224746 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:13.224331 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" event={"ID":"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65","Type":"ContainerDied","Data":"f07aa569e6d3b2b9876cf351304737e3ba3da63efd5e1ea62549fecd46570d7a"} Nov 03 22:24:14.155965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:14.155933 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:14.379188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:14.379162 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:14.467617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:14.467590 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-xks-ca\") pod \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " Nov 03 22:24:14.467798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:14.467722 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-kube-api-access-cbpn4\" (UniqueName: \"kubernetes.io/projected/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-kube-api-access-cbpn4\") pod \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " Nov 03 22:24:14.467798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:14.467747 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-openshift-ca\") pod \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\" (UID: \"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65\") " Nov 03 22:24:14.468177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:14.468141 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-openshift-ca" pod "355d6d35-1bf8-4e19-8eeb-ed6d6c026b65" (UID: "355d6d35-1bf8-4e19-8eeb-ed6d6c026b65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:24:14.469676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:14.469637 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-xks-ca" pod "355d6d35-1bf8-4e19-8eeb-ed6d6c026b65" (UID: "355d6d35-1bf8-4e19-8eeb-ed6d6c026b65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:14.469787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:14.469754 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-kube-api-access-cbpn4" pod "355d6d35-1bf8-4e19-8eeb-ed6d6c026b65" (UID: "355d6d35-1bf8-4e19-8eeb-ed6d6c026b65"). InnerVolumeSpecName "kube-api-access-cbpn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:24:14.568943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:14.568910 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbpn4\" (UniqueName: \"kubernetes.io/projected/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-kube-api-access-cbpn4\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:14.568943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:14.568937 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:14.568943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:14.568949 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:15.239382 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:15.239353 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" Nov 03 22:24:15.239546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:15.239400 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9" event={"ID":"355d6d35-1bf8-4e19-8eeb-ed6d6c026b65","Type":"ContainerDied","Data":"f6f5ffb0d229229ede85047019361dbd4ffc842d50781683d49c8ec9286e62dd"} Nov 03 22:24:15.239546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:15.239449 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6f5ffb0d229229ede85047019361dbd4ffc842d50781683d49c8ec9286e62dd" Nov 03 22:24:17.552911 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:24:17.552866 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:24:18.619793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:18.619758 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:23.125813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:23.125753 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:27.673612 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:27.673523 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:29.043898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:29.043854 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:32.279132 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:32.279095 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:32.357106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:32.357071 4298 generic.go:356] "Generic (PLEG): container finished" podID="ca4b142d-366e-4801-8597-706f1160c4ad" containerID="0170eab387af5dd918f1e7e8981f753f6555aabe56d6f12b4069e20579b40225" exitCode=1 Nov 03 22:24:32.357262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:32.357142 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" event={"ID":"ca4b142d-366e-4801-8597-706f1160c4ad","Type":"ContainerDied","Data":"0170eab387af5dd918f1e7e8981f753f6555aabe56d6f12b4069e20579b40225"} Nov 03 22:24:32.357564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:32.357550 4298 scope.go:122] "RemoveContainer" containerID="0170eab387af5dd918f1e7e8981f753f6555aabe56d6f12b4069e20579b40225" Nov 03 22:24:32.357993 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:32.357979 4298 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:24:34.248315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.248274 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:24:34.371710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.371672 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" event={"ID":"ca4b142d-366e-4801-8597-706f1160c4ad","Type":"ContainerStarted","Data":"5ee39809fe9e00c18ecf339736656e11966391af99706da20ec77ad846da28a0"} Nov 03 22:24:34.371879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.371811 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:24:34.420178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.420145 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf"] Nov 03 22:24:34.420634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.420621 4298 cpu_manager.go:414] "containerMap: removing container" podUID="355d6d35-1bf8-4e19-8eeb-ed6d6c026b65" containerName="revoke-keys" Nov 03 22:24:34.420708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.420636 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355d6d35-1bf8-4e19-8eeb-ed6d6c026b65" containerName="revoke-keys" Nov 03 22:24:34.420765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.420756 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="355d6d35-1bf8-4e19-8eeb-ed6d6c026b65" containerName="revoke-keys" Nov 03 22:24:34.424112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.424097 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.527328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.527253 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"distinct-w5-f81f8f-kserve-self-signed-certs\"" Nov 03 22:24:34.530654 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.530624 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf"] Nov 03 22:24:34.584944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.584901 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:34.596257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.596225 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-model-cache\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.596507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.596347 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tls-certs\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.596507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.596420 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-dshm\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.596507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.596455 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-home\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.596679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.596562 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz4lt\" (UniqueName: \"kubernetes.io/projected/894f31d0-02aa-445d-9ba8-8628e8d41e0e-kube-api-access-nz4lt\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.596679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.596608 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tmp-dir\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.697513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.697470 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tls-certs\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.697717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.697570 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-dshm\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.697717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.697602 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-home\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.697717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.697686 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz4lt\" (UniqueName: \"kubernetes.io/projected/894f31d0-02aa-445d-9ba8-8628e8d41e0e-kube-api-access-nz4lt\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.697913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.697742 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tmp-dir\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.697913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.697788 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-model-cache\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.698255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.698200 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tmp-dir\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.698255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.698248 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-model-cache\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.698443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.698354 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-home\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.700029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.700010 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-dshm\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.700501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.700480 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tls-certs\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:34.757144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:34.757108 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz4lt\" (UniqueName: \"kubernetes.io/projected/894f31d0-02aa-445d-9ba8-8628e8d41e0e-kube-api-access-nz4lt\") pod \"distinct-w5-f81f8f-kserve-66c55565cf-7b8qf\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:35.036396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:35.036362 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:35.407916 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:24:35.407889 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod894f31d0_02aa_445d_9ba8_8628e8d41e0e.slice/crio-2cd79a2cd2548796cba6ff7a27b9642d7449ef3c1a006e6c5837e24e3f2cafd6 WatchSource:0}: Error finding container 2cd79a2cd2548796cba6ff7a27b9642d7449ef3c1a006e6c5837e24e3f2cafd6: Status 404 returned error can't find the container with id 2cd79a2cd2548796cba6ff7a27b9642d7449ef3c1a006e6c5837e24e3f2cafd6 Nov 03 22:24:35.411182 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:35.411154 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf"] Nov 03 22:24:35.973605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:35.973569 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t"] Nov 03 22:24:35.977504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:35.977488 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.030989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.030957 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6a6b7f-openshift-default-dockercfg-fqnpk\"" Nov 03 22:24:36.034368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.034321 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t"] Nov 03 22:24:36.114140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.114108 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a156fac-b1b4-4cf1-a546-658009fec833-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.114319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.114148 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a156fac-b1b4-4cf1-a546-658009fec833-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.114319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.114251 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.114392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.114358 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.114392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.114382 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.114485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.114472 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.114532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.114493 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.114532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.114509 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jp6p\" (UniqueName: \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-kube-api-access-2jp6p\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.114595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.114535 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.215590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.215552 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.215799 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.215619 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a156fac-b1b4-4cf1-a546-658009fec833-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.215799 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.215682 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a156fac-b1b4-4cf1-a546-658009fec833-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.215799 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.215751 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.215946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.215829 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.215946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.215854 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.215946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.215924 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.216102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.215952 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.216102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.215978 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jp6p\" (UniqueName: \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-kube-api-access-2jp6p\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.216102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.216081 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.216268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.216196 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.216411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.216386 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.216475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.216461 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.216549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.216516 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a156fac-b1b4-4cf1-a546-658009fec833-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.218099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.218077 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.218334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.218315 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a156fac-b1b4-4cf1-a546-658009fec833-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.273856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.273785 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.274013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.273938 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jp6p\" (UniqueName: \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-kube-api-access-2jp6p\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.291470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.291443 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:36.386197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.386164 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" event={"ID":"894f31d0-02aa-445d-9ba8-8628e8d41e0e","Type":"ContainerStarted","Data":"7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5"} Nov 03 22:24:36.386385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.386205 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" event={"ID":"894f31d0-02aa-445d-9ba8-8628e8d41e0e","Type":"ContainerStarted","Data":"2cd79a2cd2548796cba6ff7a27b9642d7449ef3c1a006e6c5837e24e3f2cafd6"} Nov 03 22:24:36.503301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.503245 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podStartSLOduration=2.5032324790000002 podStartE2EDuration="2.503232479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:36.50218273 +0000 UTC m=+1033.482916382" watchObservedRunningTime="2026-11-03 22:24:36.503232479 +0000 UTC m=+1033.483966129" Nov 03 22:24:36.710193 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:24:36.710158 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a156fac_b1b4_4cf1_a546_658009fec833.slice/crio-823d964a58e2ae9c55a436f24912e8b6d0ad190773ce96e67771af763a8e3e6e WatchSource:0}: Error finding container 823d964a58e2ae9c55a436f24912e8b6d0ad190773ce96e67771af763a8e3e6e: Status 404 returned error can't find the container with id 823d964a58e2ae9c55a436f24912e8b6d0ad190773ce96e67771af763a8e3e6e Nov 03 22:24:36.713388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.713350 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:24:36.713478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.713439 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:24:36.713520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.713480 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:24:36.713694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.713673 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t"] Nov 03 22:24:36.904489 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:36.904446 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:37.037516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.037437 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:37.393368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.393275 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" event={"ID":"9a156fac-b1b4-4cf1-a546-658009fec833","Type":"ContainerStarted","Data":"a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f"} Nov 03 22:24:37.393368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.393333 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" event={"ID":"9a156fac-b1b4-4cf1-a546-658009fec833","Type":"ContainerStarted","Data":"823d964a58e2ae9c55a436f24912e8b6d0ad190773ce96e67771af763a8e3e6e"} Nov 03 22:24:37.503907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.503874 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-jwnml"] Nov 03 22:24:37.504385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.504151 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/odh-model-controller-769875db56-jwnml" podUID="85f834ec-6594-48d1-87b8-2cc2299860e9" containerName="manager" containerID="cri-o://6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2" gracePeriod=10 Nov 03 22:24:37.555377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.555317 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" podStartSLOduration=2.555301831 podStartE2EDuration="2.555301831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:37.552872838 +0000 UTC m=+1034.533606491" watchObservedRunningTime="2026-11-03 22:24:37.555301831 +0000 UTC m=+1034.536035484" Nov 03 22:24:37.791564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.791537 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:24:37.857098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.857060 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-cert\" (UniqueName: \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-cert\") pod \"85f834ec-6594-48d1-87b8-2cc2299860e9\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " Nov 03 22:24:37.857243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.857188 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-metrics-certs\") pod \"85f834ec-6594-48d1-87b8-2cc2299860e9\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " Nov 03 22:24:37.857321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.857255 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f834ec-6594-48d1-87b8-2cc2299860e9-kube-api-access-rq9tk\" (UniqueName: \"kubernetes.io/projected/85f834ec-6594-48d1-87b8-2cc2299860e9-kube-api-access-rq9tk\") pod \"85f834ec-6594-48d1-87b8-2cc2299860e9\" (UID: \"85f834ec-6594-48d1-87b8-2cc2299860e9\") " Nov 03 22:24:37.859305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.859277 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-metrics-certs" pod "85f834ec-6594-48d1-87b8-2cc2299860e9" (UID: "85f834ec-6594-48d1-87b8-2cc2299860e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:37.859429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.859321 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-cert" pod "85f834ec-6594-48d1-87b8-2cc2299860e9" (UID: "85f834ec-6594-48d1-87b8-2cc2299860e9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:37.859429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.859397 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f834ec-6594-48d1-87b8-2cc2299860e9-kube-api-access-rq9tk" pod "85f834ec-6594-48d1-87b8-2cc2299860e9" (UID: "85f834ec-6594-48d1-87b8-2cc2299860e9"). InnerVolumeSpecName "kube-api-access-rq9tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:24:37.958981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.958947 4298 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:37.958981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.958980 4298 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f834ec-6594-48d1-87b8-2cc2299860e9-metrics-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:37.959184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:37.958994 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq9tk\" (UniqueName: \"kubernetes.io/projected/85f834ec-6594-48d1-87b8-2cc2299860e9-kube-api-access-rq9tk\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:38.292424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.292335 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:38.294085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.294057 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:24:38.294187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.294109 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 22:24:38.400219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.400185 4298 generic.go:356] "Generic (PLEG): container finished" podID="85f834ec-6594-48d1-87b8-2cc2299860e9" containerID="6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2" exitCode=0 Nov 03 22:24:38.400398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.400227 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-jwnml" Nov 03 22:24:38.400398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.400270 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jwnml" event={"ID":"85f834ec-6594-48d1-87b8-2cc2299860e9","Type":"ContainerDied","Data":"6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2"} Nov 03 22:24:38.400398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.400337 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jwnml" event={"ID":"85f834ec-6594-48d1-87b8-2cc2299860e9","Type":"ContainerDied","Data":"01cbedd28cccbc93aa288fe4acbcba8ad78a42f477ab191fb9d0386ef48ad142"} Nov 03 22:24:38.400398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.400364 4298 scope.go:122] "RemoveContainer" containerID="6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2" Nov 03 22:24:38.411104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.411071 4298 scope.go:122] "RemoveContainer" containerID="6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2" Nov 03 22:24:38.411370 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:24:38.411349 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2\": container with ID starting with 6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2 not found: ID does not exist" containerID="6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2" Nov 03 22:24:38.411412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.411377 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2"} err="failed to get container status \"6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2\": rpc error: code = NotFound desc = could not find container \"6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2\": container with ID starting with 6df99ae233869e39360806554d595948bff7048f032c96de7a7a8530073dfff2 not found: ID does not exist" Nov 03 22:24:38.572259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.572166 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-jwnml"] Nov 03 22:24:38.577929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:38.577903 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/odh-model-controller-769875db56-jwnml"] Nov 03 22:24:39.152850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:39.152805 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f834ec-6594-48d1-87b8-2cc2299860e9" path="/var/lib/kubelet/pods/85f834ec-6594-48d1-87b8-2cc2299860e9/volumes" Nov 03 22:24:39.292339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:39.292294 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:24:39.292515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:39.292359 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 22:24:40.292426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:40.292394 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:24:40.292862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:40.292444 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 22:24:41.291828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:41.291795 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:24:41.293936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:41.291849 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 22:24:41.691121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:41.691029 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:42.292656 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:42.292614 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:24:42.292832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:42.292677 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 22:24:42.404835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:42.404799 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd"] Nov 03 22:24:42.405237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:42.404991 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" podUID="95448530-22d3-444e-abde-6b6b873f139f" containerName="istio-proxy" containerID="cri-o://f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83" gracePeriod=30 Nov 03 22:24:43.292509 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:43.292470 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:24:43.292700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:43.292530 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 22:24:44.296667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:44.296623 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:44.297169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:44.297148 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:44.297890 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:44.297866 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:24:45.037186 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:45.037152 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:45.049779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:45.049756 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:45.379908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:45.379832 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-4p5nb" Nov 03 22:24:45.486742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:45.486713 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:24:46.211061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:46.211027 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:47.553383 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:24:47.553331 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:24:47.675990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.675934 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:24:47.770151 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770110 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-credential-socket\") pod \"95448530-22d3-444e-abde-6b6b873f139f\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " Nov 03 22:24:47.770318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770218 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-envoy\") pod \"95448530-22d3-444e-abde-6b6b873f139f\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " Nov 03 22:24:47.770318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770282 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-certs\") pod \"95448530-22d3-444e-abde-6b6b873f139f\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " Nov 03 22:24:47.770424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770324 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-istio-token\" (UniqueName: \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-istio-token\") pod \"95448530-22d3-444e-abde-6b6b873f139f\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " Nov 03 22:24:47.770424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770367 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-credential-socket" pod "95448530-22d3-444e-abde-6b6b873f139f" (UID: "95448530-22d3-444e-abde-6b6b873f139f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:47.770424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770383 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-kube-api-access-vk2t6\" (UniqueName: \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-kube-api-access-vk2t6\") pod \"95448530-22d3-444e-abde-6b6b873f139f\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " Nov 03 22:24:47.770590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770424 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95448530-22d3-444e-abde-6b6b873f139f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95448530-22d3-444e-abde-6b6b873f139f-istio-podinfo\") pod \"95448530-22d3-444e-abde-6b6b873f139f\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " Nov 03 22:24:47.770590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770461 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-socket\") pod \"95448530-22d3-444e-abde-6b6b873f139f\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " Nov 03 22:24:47.770590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770493 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95448530-22d3-444e-abde-6b6b873f139f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95448530-22d3-444e-abde-6b6b873f139f-istiod-ca-cert\") pod \"95448530-22d3-444e-abde-6b6b873f139f\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " Nov 03 22:24:47.770794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770609 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-data\") pod \"95448530-22d3-444e-abde-6b6b873f139f\" (UID: \"95448530-22d3-444e-abde-6b6b873f139f\") " Nov 03 22:24:47.770794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770628 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-certs" pod "95448530-22d3-444e-abde-6b6b873f139f" (UID: "95448530-22d3-444e-abde-6b6b873f139f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:47.770904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770867 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-socket" pod "95448530-22d3-444e-abde-6b6b873f139f" (UID: "95448530-22d3-444e-abde-6b6b873f139f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:47.771013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.770984 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95448530-22d3-444e-abde-6b6b873f139f-istiod-ca-cert" pod "95448530-22d3-444e-abde-6b6b873f139f" (UID: "95448530-22d3-444e-abde-6b6b873f139f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:24:47.771109 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.771083 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-data" pod "95448530-22d3-444e-abde-6b6b873f139f" (UID: "95448530-22d3-444e-abde-6b6b873f139f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:47.771155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.771107 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:47.771155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.771128 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:47.771155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.771142 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:47.773259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.773233 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-istio-token" pod "95448530-22d3-444e-abde-6b6b873f139f" (UID: "95448530-22d3-444e-abde-6b6b873f139f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:24:47.773368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.773306 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-envoy" pod "95448530-22d3-444e-abde-6b6b873f139f" (UID: "95448530-22d3-444e-abde-6b6b873f139f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:47.773443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.773402 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-kube-api-access-vk2t6" pod "95448530-22d3-444e-abde-6b6b873f139f" (UID: "95448530-22d3-444e-abde-6b6b873f139f"). InnerVolumeSpecName "kube-api-access-vk2t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:24:47.773730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.773710 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95448530-22d3-444e-abde-6b6b873f139f-istio-podinfo" pod "95448530-22d3-444e-abde-6b6b873f139f" (UID: "95448530-22d3-444e-abde-6b6b873f139f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:24:47.872234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.872197 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95448530-22d3-444e-abde-6b6b873f139f-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:47.872234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.872227 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95448530-22d3-444e-abde-6b6b873f139f-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:47.872234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.872238 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:47.872524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.872246 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95448530-22d3-444e-abde-6b6b873f139f-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:47.872524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.872255 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:47.872524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:47.872263 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk2t6\" (UniqueName: \"kubernetes.io/projected/95448530-22d3-444e-abde-6b6b873f139f-kube-api-access-vk2t6\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:48.495538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:48.495505 4298 generic.go:356] "Generic (PLEG): container finished" podID="95448530-22d3-444e-abde-6b6b873f139f" containerID="f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83" exitCode=0 Nov 03 22:24:48.495725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:48.495555 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" Nov 03 22:24:48.495725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:48.495584 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" event={"ID":"95448530-22d3-444e-abde-6b6b873f139f","Type":"ContainerDied","Data":"f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83"} Nov 03 22:24:48.495725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:48.495637 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd" event={"ID":"95448530-22d3-444e-abde-6b6b873f139f","Type":"ContainerDied","Data":"49b615ba27ac043a8130bb31b8861b4c34cf449e6752d39eba38003066f1f23a"} Nov 03 22:24:48.495725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:48.495668 4298 scope.go:122] "RemoveContainer" containerID="f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83" Nov 03 22:24:48.505693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:48.505670 4298 scope.go:122] "RemoveContainer" containerID="f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83" Nov 03 22:24:48.505947 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:24:48.505928 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83\": container with ID starting with f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83 not found: ID does not exist" containerID="f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83" Nov 03 22:24:48.506021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:48.505957 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83"} err="failed to get container status \"f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83\": rpc error: code = NotFound desc = could not find container \"f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83\": container with ID starting with f635e34ce468ce16e4cf8315e6127b5c926c4cc3fa175618455648d0a21bca83 not found: ID does not exist" Nov 03 22:24:48.720538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:48.720503 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd"] Nov 03 22:24:48.771487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:48.771405 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4szjd"] Nov 03 22:24:49.134902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:49.134829 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95448530-22d3-444e-abde-6b6b873f139f" path="/var/lib/kubelet/pods/95448530-22d3-444e-abde-6b6b873f139f/volumes" Nov 03 22:24:50.752927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:50.752892 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:55.278626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:55.278585 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:59.784957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:24:59.784918 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:04.277509 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:04.277474 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:08.478563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:08.478528 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:08.788940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:08.788847 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:11.724685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:11.724637 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:11.898422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:11.898391 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:12.930515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:12.930485 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:13.381889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:13.381854 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:17.552046 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:25:17.552009 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:25:17.892199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:17.892119 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:19.460179 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.460146 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6"] Nov 03 22:25:19.460678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.460663 4298 cpu_manager.go:414] "containerMap: removing container" podUID="95448530-22d3-444e-abde-6b6b873f139f" containerName="istio-proxy" Nov 03 22:25:19.460725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.460678 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95448530-22d3-444e-abde-6b6b873f139f" containerName="istio-proxy" Nov 03 22:25:19.460725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.460713 4298 cpu_manager.go:414] "containerMap: removing container" podUID="85f834ec-6594-48d1-87b8-2cc2299860e9" containerName="manager" Nov 03 22:25:19.460725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.460718 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f834ec-6594-48d1-87b8-2cc2299860e9" containerName="manager" Nov 03 22:25:19.460821 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.460788 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f834ec-6594-48d1-87b8-2cc2299860e9" containerName="manager" Nov 03 22:25:19.460821 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.460798 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="95448530-22d3-444e-abde-6b6b873f139f" containerName="istio-proxy" Nov 03 22:25:19.464798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.464780 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.520768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.520731 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd50b56c-786a-48d9-a89a-431ab176d151-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.520768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.520771 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2622k\" (UniqueName: \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-kube-api-access-2622k\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.520971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.520791 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.521015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.520956 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd50b56c-786a-48d9-a89a-431ab176d151-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.521056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.521020 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.521090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.521074 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.521147 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.521131 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.521244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.521221 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.521315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.521298 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.557967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.557939 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6"] Nov 03 22:25:19.622898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.622866 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.622920 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd50b56c-786a-48d9-a89a-431ab176d151-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.622940 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2622k\" (UniqueName: \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-kube-api-access-2622k\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.622956 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.622987 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd50b56c-786a-48d9-a89a-431ab176d151-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.623020 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.623043 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.623085 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.623153 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.623337 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.623430 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.623520 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.623554 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.623704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.623686 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd50b56c-786a-48d9-a89a-431ab176d151-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.625438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.625417 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.625578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.625562 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd50b56c-786a-48d9-a89a-431ab176d151-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.680465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.680431 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.680589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.680507 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2622k\" (UniqueName: \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-kube-api-access-2622k\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:19.779666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:19.779555 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:20.201059 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:25:20.201031 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd50b56c_786a_48d9_a89a_431ab176d151.slice/crio-fe56140fa8ade8d2985105fa28badd248b3df2f53a1edabb3998bc05faed9a7c WatchSource:0}: Error finding container fe56140fa8ade8d2985105fa28badd248b3df2f53a1edabb3998bc05faed9a7c: Status 404 returned error can't find the container with id fe56140fa8ade8d2985105fa28badd248b3df2f53a1edabb3998bc05faed9a7c Nov 03 22:25:20.203752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:20.203724 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:25:20.203844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:20.203790 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:25:20.203844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:20.203825 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:25:20.205145 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:20.205122 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6"] Nov 03 22:25:20.695357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:20.695322 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" event={"ID":"cd50b56c-786a-48d9-a89a-431ab176d151","Type":"ContainerStarted","Data":"8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89"} Nov 03 22:25:20.695772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:20.695381 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" event={"ID":"cd50b56c-786a-48d9-a89a-431ab176d151","Type":"ContainerStarted","Data":"fe56140fa8ade8d2985105fa28badd248b3df2f53a1edabb3998bc05faed9a7c"} Nov 03 22:25:20.780357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:20.780326 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:20.813112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:20.813053 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" podStartSLOduration=1.813040094 podStartE2EDuration="1.813040094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:20.812521177 +0000 UTC m=+1077.793254830" watchObservedRunningTime="2026-11-03 22:25:20.813040094 +0000 UTC m=+1077.793773747" Nov 03 22:25:20.854425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:20.854394 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:21.781914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:21.781876 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:25:21.782390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:21.781958 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 22:25:22.394200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:22.394166 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:22.780846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:22.780816 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:25:22.781200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:22.781169 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 22:25:23.780582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:23.780547 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:25:23.780992 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:23.780600 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 22:25:24.779858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:24.779820 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:25:24.780049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:24.779887 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 22:25:25.780802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:25.780771 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:25:25.781243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:25.780848 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 22:25:26.780129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:26.780094 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:25:26.780303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:26.780148 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 22:25:26.827718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:26.827682 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:27.784591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:27.784565 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:27.785048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:27.785027 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:27.785566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:27.785550 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:25:31.363501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:31.363433 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:35.905401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:35.905360 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:40.395201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:40.395164 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:44.886094 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:44.886064 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:47.552880 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:25:47.552841 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:25:49.363372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:49.363339 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:53.889058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:53.889011 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:56.739406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:56.739364 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:58.397513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:25:58.397446 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:02.902510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:02.902460 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:06.079894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.079859 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml"] Nov 03 22:26:06.084219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.084202 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.172699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.172669 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml"] Nov 03 22:26:06.200703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.200662 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.200919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.200728 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.200919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.200885 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.201015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.200983 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.201015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.201006 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.201093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.201075 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.201128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.201108 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.201172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.201161 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.201209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.201189 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cld42\" (UniqueName: \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-kube-api-access-cld42\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.302837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.302798 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.302837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.302842 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.302866 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.302979 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.303040 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.303070 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cld42\" (UniqueName: \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-kube-api-access-cld42\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.303122 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.303181 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.303232 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.303302 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.303716 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303861 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.303774 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303861 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.303846 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.303961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.303942 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.305371 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.305338 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.305620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.305601 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.366154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.366074 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.366311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.366195 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cld42\" (UniqueName: \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-kube-api-access-cld42\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.397920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.397881 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:06.467349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.467302 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:06.819171 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:26:06.819135 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927ec6e4_c00c_4cf7_a1c6_8317f9552a95.slice/crio-537ef1904a01b2bccd1a21227edb5fddd1c5a923e3c5a29e322dcb762253af86 WatchSource:0}: Error finding container 537ef1904a01b2bccd1a21227edb5fddd1c5a923e3c5a29e322dcb762253af86: Status 404 returned error can't find the container with id 537ef1904a01b2bccd1a21227edb5fddd1c5a923e3c5a29e322dcb762253af86 Nov 03 22:26:06.822206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.822164 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:26:06.822290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.822258 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:26:06.822329 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.822299 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:26:06.822605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:06.822582 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml"] Nov 03 22:26:07.008434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:07.008397 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" event={"ID":"927ec6e4-c00c-4cf7-a1c6-8317f9552a95","Type":"ContainerStarted","Data":"666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d"} Nov 03 22:26:07.008635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:07.008460 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" event={"ID":"927ec6e4-c00c-4cf7-a1c6-8317f9552a95","Type":"ContainerStarted","Data":"537ef1904a01b2bccd1a21227edb5fddd1c5a923e3c5a29e322dcb762253af86"} Nov 03 22:26:07.129918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:07.129800 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" podStartSLOduration=1.129782655 podStartE2EDuration="1.129782655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:07.126062008 +0000 UTC m=+1124.106795662" watchObservedRunningTime="2026-11-03 22:26:07.129782655 +0000 UTC m=+1124.110516307" Nov 03 22:26:07.400464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:07.400371 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:07.400809 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:07.400780 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:26:07.401013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:07.400987 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:26:07.517962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:07.517922 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:08.398786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:08.398749 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:26:08.399191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:08.398803 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:26:08.848668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:08.840054 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:09.399129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:09.399085 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:26:09.399675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:09.399143 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:26:10.399187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:10.399153 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:26:10.399620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:10.399207 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:26:11.399276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:11.399241 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:26:11.399700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:11.399295 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:26:11.997915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:11.997886 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:12.399248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.399167 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:26:12.399248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.399221 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:26:12.582044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.582014 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl"] Nov 03 22:26:12.585815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.585799 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.657511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.657425 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl"] Nov 03 22:26:12.696937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.696907 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-model-cache\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.697117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.696949 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f816022-7875-4505-99ff-072c84a5dc79-tls-certs\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.697117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.697036 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-tmp-dir\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.697117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.697108 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-dshm\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.697235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.697128 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c228\" (UniqueName: \"kubernetes.io/projected/0f816022-7875-4505-99ff-072c84a5dc79-kube-api-access-8c228\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.697319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.697303 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-home\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.700797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.700773 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"distinct-2-w6-6a6b7f-kserve-self-signed-certs\"" Nov 03 22:26:12.706227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.706177 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:12.798180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.798142 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f816022-7875-4505-99ff-072c84a5dc79-tls-certs\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.798378 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.798226 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-tmp-dir\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.798378 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.798272 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-dshm\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.798378 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.798335 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c228\" (UniqueName: \"kubernetes.io/projected/0f816022-7875-4505-99ff-072c84a5dc79-kube-api-access-8c228\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.798569 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.798551 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-home\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.798634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.798616 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-model-cache\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.798750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.798724 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-tmp-dir\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.799038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.798990 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-model-cache\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.799255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.799233 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-home\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.804683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.804634 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f816022-7875-4505-99ff-072c84a5dc79-tls-certs\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.804906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.804885 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-dshm\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.859161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.859128 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c228\" (UniqueName: \"kubernetes.io/projected/0f816022-7875-4505-99ff-072c84a5dc79-kube-api-access-8c228\") pod \"distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:12.897935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:12.897903 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:13.323590 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:26:13.323555 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f816022_7875_4505_99ff_072c84a5dc79.slice/crio-10b19c889fd5dd1ce871fd1f9d090a0b0c3de41a9b9226d228de45f8e416b261 WatchSource:0}: Error finding container 10b19c889fd5dd1ce871fd1f9d090a0b0c3de41a9b9226d228de45f8e416b261: Status 404 returned error can't find the container with id 10b19c889fd5dd1ce871fd1f9d090a0b0c3de41a9b9226d228de45f8e416b261 Nov 03 22:26:13.326590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:13.326568 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl"] Nov 03 22:26:13.398898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:13.398872 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:26:13.399021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:13.398913 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:26:14.085412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:14.085368 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" event={"ID":"0f816022-7875-4505-99ff-072c84a5dc79","Type":"ContainerStarted","Data":"ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e"} Nov 03 22:26:14.085822 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:14.085420 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" event={"ID":"0f816022-7875-4505-99ff-072c84a5dc79","Type":"ContainerStarted","Data":"10b19c889fd5dd1ce871fd1f9d090a0b0c3de41a9b9226d228de45f8e416b261"} Nov 03 22:26:14.201204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:14.201157 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podStartSLOduration=2.201142829 podStartE2EDuration="2.201142829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:14.201003749 +0000 UTC m=+1131.181737403" watchObservedRunningTime="2026-11-03 22:26:14.201142829 +0000 UTC m=+1131.181876478" Nov 03 22:26:14.403299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:14.403220 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:14.403623 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:14.403602 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:14.404342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:14.404321 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:26:14.899074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:14.899036 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:16.505668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:16.505519 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:17.552452 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:26:17.552409 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:26:19.134327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:19.134299 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:19.304377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:19.304347 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:20.986703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:20.986668 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:22.898217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:22.898186 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:22.911237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:22.911212 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:23.155270 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:23.155188 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:26:25.606376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:25.606335 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:30.099531 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:30.099495 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:34.593779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:34.593736 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:39.053659 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:39.053607 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj"] Nov 03 22:26:39.105931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:39.105893 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a3a810-6xdmj"] Nov 03 22:26:39.135716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:39.135682 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af98e84-a10c-4119-a6e9-c9ceda7e8c96" path="/var/lib/kubelet/pods/9af98e84-a10c-4119-a6e9-c9ceda7e8c96/volumes" Nov 03 22:26:39.203346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:39.203312 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:43.789863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:43.789828 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:47.552954 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:26:47.552914 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:26:48.277160 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:48.277127 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:50.011985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:50.011948 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:52.783269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:52.783212 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:57.395755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:26:57.395717 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:01.316533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.316486 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km"] Nov 03 22:27:01.325338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.325309 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.372292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.372259 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km"] Nov 03 22:27:01.418410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.418380 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"distinct-2-w5-f81f8f-kserve-self-signed-certs\"" Nov 03 22:27:01.425708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.425680 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-dshm\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.425792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.425721 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-model-cache\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.425859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.425838 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tls-certs\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.425908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.425893 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztmg\" (UniqueName: \"kubernetes.io/projected/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-kube-api-access-pztmg\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.425976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.425964 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-home\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.426017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.425995 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tmp-dir\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.479614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.479575 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:01.527062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.527025 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztmg\" (UniqueName: \"kubernetes.io/projected/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-kube-api-access-pztmg\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.527277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.527115 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-home\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.527277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.527140 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tmp-dir\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.527277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.527244 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-dshm\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.527450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.527281 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-model-cache\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.527450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.527330 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tls-certs\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.527738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.527604 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-home\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.527738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.527685 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tmp-dir\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.527864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.527762 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-model-cache\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.529980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.529950 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-dshm\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.530336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.530313 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tls-certs\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.591183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.591105 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztmg\" (UniqueName: \"kubernetes.io/projected/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-kube-api-access-pztmg\") pod \"distinct-2-w5-f81f8f-kserve-69bf799d56-g66km\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.637516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.637477 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:01.920189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:01.920062 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:02.021046 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:27:02.021017 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cd70cf3_1d97_4c3a_9ada_bed7517ca13f.slice/crio-b42e80b24f064ec59af298795308a30cbd44057ad24838dbe03c8640df5c6a3c WatchSource:0}: Error finding container b42e80b24f064ec59af298795308a30cbd44057ad24838dbe03c8640df5c6a3c: Status 404 returned error can't find the container with id b42e80b24f064ec59af298795308a30cbd44057ad24838dbe03c8640df5c6a3c Nov 03 22:27:02.024694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:02.024668 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km"] Nov 03 22:27:02.410375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:02.410341 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" event={"ID":"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f","Type":"ContainerStarted","Data":"0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8"} Nov 03 22:27:02.410375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:02.410382 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" event={"ID":"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f","Type":"ContainerStarted","Data":"b42e80b24f064ec59af298795308a30cbd44057ad24838dbe03c8640df5c6a3c"} Nov 03 22:27:02.527097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:02.527043 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podStartSLOduration=1.5270276790000001 podStartE2EDuration="1.527027679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:02.525000028 +0000 UTC m=+1179.505733681" watchObservedRunningTime="2026-11-03 22:27:02.527027679 +0000 UTC m=+1179.507761331" Nov 03 22:27:02.637591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:02.637551 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:04.397331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:04.397292 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:06.415638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:06.415602 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:07.166791 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.166756 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k"] Nov 03 22:27:07.170990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.170966 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.199690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.199633 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.199873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.199796 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.199873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.199826 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.199873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.199847 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.200033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.199965 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.200118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.200096 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914760e1-f094-48ab-848f-e0e168fb265e-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.200206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.200191 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914760e1-f094-48ab-848f-e0e168fb265e-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.200297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.200283 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.200360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.200317 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d948h\" (UniqueName: \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-kube-api-access-d948h\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.278984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.278947 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k"] Nov 03 22:27:07.302022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.301986 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302022 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302043 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302082 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302133 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914760e1-f094-48ab-848f-e0e168fb265e-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302207 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914760e1-f094-48ab-848f-e0e168fb265e-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302250 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302273 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d948h\" (UniqueName: \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-kube-api-access-d948h\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302601 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302332 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302601 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302387 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302601 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302562 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302619 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.302774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302759 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.303000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.302977 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914760e1-f094-48ab-848f-e0e168fb265e-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.304502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.304477 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.305069 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.305045 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914760e1-f094-48ab-848f-e0e168fb265e-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.360615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.360576 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.360829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.360738 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d948h\" (UniqueName: \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-kube-api-access-d948h\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.485796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.485758 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:07.906203 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:27:07.906173 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914760e1_f094_48ab_848f_e0e168fb265e.slice/crio-00fe43e61847c17a8f9368798ddc62728ef759332a1d2538cb2624f63354a344 WatchSource:0}: Error finding container 00fe43e61847c17a8f9368798ddc62728ef759332a1d2538cb2624f63354a344: Status 404 returned error can't find the container with id 00fe43e61847c17a8f9368798ddc62728ef759332a1d2538cb2624f63354a344 Nov 03 22:27:07.909174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.909142 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:27:07.909245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.909210 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:27:07.909245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.909240 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:27:07.910364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:07.910342 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k"] Nov 03 22:27:08.454439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:08.454404 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" event={"ID":"914760e1-f094-48ab-848f-e0e168fb265e","Type":"ContainerStarted","Data":"ce6967efc6f1800252af16607d97b4e8f0a86e6359eb4dcd6b7d91c58515be3c"} Nov 03 22:27:08.454665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:08.454459 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" event={"ID":"914760e1-f094-48ab-848f-e0e168fb265e","Type":"ContainerStarted","Data":"00fe43e61847c17a8f9368798ddc62728ef759332a1d2538cb2624f63354a344"} Nov 03 22:27:08.486714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:08.486678 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:08.488402 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:08.488380 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:08.488470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:08.488427 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:27:09.050301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:09.050233 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" podStartSLOduration=2.050211154 podStartE2EDuration="2.050211154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:08.610689434 +0000 UTC m=+1185.591423088" watchObservedRunningTime="2026-11-03 22:27:09.050211154 +0000 UTC m=+1186.030944810" Nov 03 22:27:09.052273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:09.052242 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd"] Nov 03 22:27:09.055650 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:09.055620 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92525c10-nj5sd"] Nov 03 22:27:09.141389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:09.141352 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427f0a31-7fb5-46a2-b5df-f9e56817abae" path="/var/lib/kubelet/pods/427f0a31-7fb5-46a2-b5df-f9e56817abae/volumes" Nov 03 22:27:09.186150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:09.186118 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:09.355370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:09.355281 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:09.486493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:09.486460 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:09.486690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:09.486523 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:27:10.486995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:10.486961 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:10.487485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:10.487012 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:27:10.871360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:10.871272 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:11.487117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:11.487083 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:11.487515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:11.487137 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:27:11.643666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:11.639714 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:11.668183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:11.668145 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:12.487962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:12.487928 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:12.488399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:12.487990 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:27:12.494496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:12.494469 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:27:13.486716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:13.486679 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:13.486915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:13.486738 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:27:14.486860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:14.486829 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:14.487283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:14.486881 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:27:15.374096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:15.374041 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:15.491635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:15.491601 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:15.492116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:15.492102 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:15.492736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:15.492717 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:27:17.552906 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:27:17.552860 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:27:19.049090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:19.049055 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98"] Nov 03 22:27:19.100134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:19.100096 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7e77a1a-nzk98"] Nov 03 22:27:19.135154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:19.135122 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc134e6-0180-46bf-a9bf-b6136a5af0c9" path="/var/lib/kubelet/pods/afc134e6-0180-46bf-a9bf-b6136a5af0c9/volumes" Nov 03 22:27:19.864004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:19.863973 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:24.362858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:24.362826 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:25.046841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:25.046807 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl"] Nov 03 22:27:25.097680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:25.097634 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c5bae8-qn4cl"] Nov 03 22:27:25.135228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:25.135198 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1cb965-343f-4eeb-9aac-696a86f6df38" path="/var/lib/kubelet/pods/ba1cb965-343f-4eeb-9aac-696a86f6df38/volumes" Nov 03 22:27:28.885345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:28.885291 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:32.907100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:32.907057 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:33.577360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:33.577320 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:36.047767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.047733 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6"] Nov 03 22:27:36.053204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.053184 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.065630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.065605 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.065761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.065675 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.065834 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.065777 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.065834 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.065817 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.065956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.065896 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-292tk\" (UniqueName: \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-kube-api-access-292tk\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.066035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.066017 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.066093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.066078 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.066155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.066140 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.066272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.066259 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.157753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.157720 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f61714-openshift-default-dockercfg-wvcgf\"" Nov 03 22:27:36.163850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.163819 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6"] Nov 03 22:27:36.167800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.167778 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.167917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.167823 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.167917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.167889 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.167989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.167922 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.167989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.167966 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.168066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.167991 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.168066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.168012 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.168066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.168049 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-292tk\" (UniqueName: \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-kube-api-access-292tk\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.168193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.168118 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.168398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.168368 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.168398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.168388 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.168592 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.168462 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.168863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.168845 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.169030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.169011 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.171460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.171414 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.172585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.172557 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.226279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.226246 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.226543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.226518 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-292tk\" (UniqueName: \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-kube-api-access-292tk\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.367225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.367129 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:36.793528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.793490 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6"] Nov 03 22:27:36.823918 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:27:36.823885 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992ba108_76e1_45ee_b87b_7fe5dcd799b5.slice/crio-8dede78b6cdf17c3f475c365b73fd97f5699612bd25f264e5aa69f18ce80b7c6 WatchSource:0}: Error finding container 8dede78b6cdf17c3f475c365b73fd97f5699612bd25f264e5aa69f18ce80b7c6: Status 404 returned error can't find the container with id 8dede78b6cdf17c3f475c365b73fd97f5699612bd25f264e5aa69f18ce80b7c6 Nov 03 22:27:36.826292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.826259 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:27:36.826379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.826328 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:27:36.826379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:36.826356 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:27:37.655808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:37.655766 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" event={"ID":"992ba108-76e1-45ee-b87b-7fe5dcd799b5","Type":"ContainerStarted","Data":"a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7"} Nov 03 22:27:37.656322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:37.655875 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" event={"ID":"992ba108-76e1-45ee-b87b-7fe5dcd799b5","Type":"ContainerStarted","Data":"8dede78b6cdf17c3f475c365b73fd97f5699612bd25f264e5aa69f18ce80b7c6"} Nov 03 22:27:37.849943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:37.849889 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" podStartSLOduration=1.849877224 podStartE2EDuration="1.849877224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:37.849412826 +0000 UTC m=+1214.830146479" watchObservedRunningTime="2026-11-03 22:27:37.849877224 +0000 UTC m=+1214.830610877" Nov 03 22:27:38.192793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:38.192762 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:38.368127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:38.368092 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:38.369672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:38.369620 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:38.369802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:38.369690 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 22:27:39.053490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:39.053448 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9"] Nov 03 22:27:39.103953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:39.103919 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf616b2-vvgc9"] Nov 03 22:27:39.156323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:39.156283 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1771fe6d-7213-4e5d-8875-311183598d2e" path="/var/lib/kubelet/pods/1771fe6d-7213-4e5d-8875-311183598d2e/volumes" Nov 03 22:27:39.368110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:39.368032 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:39.368110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:39.368086 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 22:27:40.367869 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:40.367831 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:40.368385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:40.367908 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 22:27:41.368496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:41.368455 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:41.368912 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:41.368518 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 22:27:42.368353 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:42.368308 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:42.368543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:42.368375 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 22:27:42.690594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:42.690519 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:43.367732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:43.367693 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:43.367920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:43.367750 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 22:27:44.372176 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:44.372146 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:44.372589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:44.372483 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:44.373228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:44.373208 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:27:46.784198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:46.784167 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:46.960403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:46.960374 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:47.188333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:47.188235 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:47.552166 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:27:47.552123 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:27:50.049920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:50.049883 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv"] Nov 03 22:27:50.100778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:50.100743 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6384fbd5-29mcv"] Nov 03 22:27:51.141914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:51.141874 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8444b5-9469-4f90-9f7b-6d2d44d69c75" path="/var/lib/kubelet/pods/cc8444b5-9469-4f90-9f7b-6d2d44d69c75/volumes" Nov 03 22:27:51.798377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:51.798336 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:54.014854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:54.014821 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:56.544255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:27:56.544216 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:01.083488 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:01.083445 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:02.050669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:02.050623 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2"] Nov 03 22:28:02.100117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:02.100080 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4d8d7b-zc7w2"] Nov 03 22:28:03.135396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:03.135361 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b92ffa9-fc55-4933-a64f-9348bfd8e18a" path="/var/lib/kubelet/pods/4b92ffa9-fc55-4933-a64f-9348bfd8e18a/volumes" Nov 03 22:28:05.575588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:05.575540 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:06.382277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:06.382236 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:07.047804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.047769 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb"] Nov 03 22:28:07.098802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.098771 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4171f1ab-7gkzb"] Nov 03 22:28:07.135398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.135369 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9d9c7d-0245-4063-8d06-5b2384bc538d" path="/var/lib/kubelet/pods/1b9d9c7d-0245-4063-8d06-5b2384bc538d/volumes" Nov 03 22:28:07.712682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.712631 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws"] Nov 03 22:28:07.717901 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.717883 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:07.769787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.769752 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws"] Nov 03 22:28:07.817024 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.816996 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:07.817233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.817215 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:28:07.835403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.835374 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64455612-fx2ws\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:07.835403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.835405 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc6fv\" (UniqueName: \"kubernetes.io/projected/777d8e0a-571f-4941-b6e6-45aa67036eff-kube-api-access-lc6fv\") pod \"maas-api-revoke-sub-64455612-fx2ws\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:07.835629 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.835609 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64455612-fx2ws\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:07.937380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.937340 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64455612-fx2ws\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:07.937380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.937380 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc6fv\" (UniqueName: \"kubernetes.io/projected/777d8e0a-571f-4941-b6e6-45aa67036eff-kube-api-access-lc6fv\") pod \"maas-api-revoke-sub-64455612-fx2ws\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:07.937625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.937607 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64455612-fx2ws\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:07.938148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.938127 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64455612-fx2ws\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:07.939993 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.939969 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64455612-fx2ws\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:07.995662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:07.995572 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc6fv\" (UniqueName: \"kubernetes.io/projected/777d8e0a-571f-4941-b6e6-45aa67036eff-kube-api-access-lc6fv\") pod \"maas-api-revoke-sub-64455612-fx2ws\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:08.031397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:08.031362 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:08.399571 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:28:08.399540 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777d8e0a_571f_4941_b6e6_45aa67036eff.slice/crio-6fb6145a2144dc55c31bc0f87eddbb3074cf5bef737f4fbea5d298a7de38463e WatchSource:0}: Error finding container 6fb6145a2144dc55c31bc0f87eddbb3074cf5bef737f4fbea5d298a7de38463e: Status 404 returned error can't find the container with id 6fb6145a2144dc55c31bc0f87eddbb3074cf5bef737f4fbea5d298a7de38463e Nov 03 22:28:08.402555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:08.402533 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws"] Nov 03 22:28:08.869386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:08.869347 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" event={"ID":"777d8e0a-571f-4941-b6e6-45aa67036eff","Type":"ContainerStarted","Data":"53aa39b8164012354b25d79c0f9c60ad8831ffb89a48f24523023d03bff931f3"} Nov 03 22:28:08.869569 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:08.869440 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" event={"ID":"777d8e0a-571f-4941-b6e6-45aa67036eff","Type":"ContainerStarted","Data":"6fb6145a2144dc55c31bc0f87eddbb3074cf5bef737f4fbea5d298a7de38463e"} Nov 03 22:28:08.986404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:08.986302 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" podStartSLOduration=1.986287275 podStartE2EDuration="1.986287275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:08.984778154 +0000 UTC m=+1245.965511806" watchObservedRunningTime="2026-11-03 22:28:08.986287275 +0000 UTC m=+1245.967020928" Nov 03 22:28:09.539123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:09.539093 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:09.877330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:09.877293 4298 generic.go:356] "Generic (PLEG): container finished" podID="777d8e0a-571f-4941-b6e6-45aa67036eff" containerID="53aa39b8164012354b25d79c0f9c60ad8831ffb89a48f24523023d03bff931f3" exitCode=0 Nov 03 22:28:09.877330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:09.877333 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" event={"ID":"777d8e0a-571f-4941-b6e6-45aa67036eff","Type":"ContainerDied","Data":"53aa39b8164012354b25d79c0f9c60ad8831ffb89a48f24523023d03bff931f3"} Nov 03 22:28:10.065495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:10.065462 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:11.033942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.033916 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:11.176959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.176873 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777d8e0a-571f-4941-b6e6-45aa67036eff-kube-api-access-lc6fv\" (UniqueName: \"kubernetes.io/projected/777d8e0a-571f-4941-b6e6-45aa67036eff-kube-api-access-lc6fv\") pod \"777d8e0a-571f-4941-b6e6-45aa67036eff\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " Nov 03 22:28:11.176959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.176916 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-xks-ca\") pod \"777d8e0a-571f-4941-b6e6-45aa67036eff\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " Nov 03 22:28:11.177180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.177016 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-openshift-ca\") pod \"777d8e0a-571f-4941-b6e6-45aa67036eff\" (UID: \"777d8e0a-571f-4941-b6e6-45aa67036eff\") " Nov 03 22:28:11.177455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.177414 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-openshift-ca" pod "777d8e0a-571f-4941-b6e6-45aa67036eff" (UID: "777d8e0a-571f-4941-b6e6-45aa67036eff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:11.177865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.177839 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:11.179431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.179401 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-xks-ca" pod "777d8e0a-571f-4941-b6e6-45aa67036eff" (UID: "777d8e0a-571f-4941-b6e6-45aa67036eff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:11.179431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.179407 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777d8e0a-571f-4941-b6e6-45aa67036eff-kube-api-access-lc6fv" pod "777d8e0a-571f-4941-b6e6-45aa67036eff" (UID: "777d8e0a-571f-4941-b6e6-45aa67036eff"). InnerVolumeSpecName "kube-api-access-lc6fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:11.279737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.279687 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc6fv\" (UniqueName: \"kubernetes.io/projected/777d8e0a-571f-4941-b6e6-45aa67036eff-kube-api-access-lc6fv\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:11.279920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.279854 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777d8e0a-571f-4941-b6e6-45aa67036eff-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:11.891369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.891338 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" Nov 03 22:28:11.891565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.891380 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws" event={"ID":"777d8e0a-571f-4941-b6e6-45aa67036eff","Type":"ContainerDied","Data":"6fb6145a2144dc55c31bc0f87eddbb3074cf5bef737f4fbea5d298a7de38463e"} Nov 03 22:28:11.891565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:11.891436 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fb6145a2144dc55c31bc0f87eddbb3074cf5bef737f4fbea5d298a7de38463e" Nov 03 22:28:13.462550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.462514 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb"] Nov 03 22:28:13.463066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.463051 4298 cpu_manager.go:414] "containerMap: removing container" podUID="777d8e0a-571f-4941-b6e6-45aa67036eff" containerName="revoke-keys" Nov 03 22:28:13.463109 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.463066 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777d8e0a-571f-4941-b6e6-45aa67036eff" containerName="revoke-keys" Nov 03 22:28:13.463166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.463157 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="777d8e0a-571f-4941-b6e6-45aa67036eff" containerName="revoke-keys" Nov 03 22:28:13.466590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.466570 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:13.521243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.521208 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb"] Nov 03 22:28:13.567190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.567158 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:13.567359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.567207 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:28:13.607879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.607844 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwpcm\" (UniqueName: \"kubernetes.io/projected/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-kube-api-access-fwpcm\") pod \"maas-api-revoke-sub-56417af0-vfztb\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:13.608062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.607886 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56417af0-vfztb\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:13.608121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.608048 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56417af0-vfztb\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:13.709429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.709396 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpcm\" (UniqueName: \"kubernetes.io/projected/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-kube-api-access-fwpcm\") pod \"maas-api-revoke-sub-56417af0-vfztb\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:13.709429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.709440 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56417af0-vfztb\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:13.709753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.709520 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56417af0-vfztb\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:13.710237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.710211 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56417af0-vfztb\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:13.712000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.711982 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56417af0-vfztb\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:13.767721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.767634 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwpcm\" (UniqueName: \"kubernetes.io/projected/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-kube-api-access-fwpcm\") pod \"maas-api-revoke-sub-56417af0-vfztb\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:13.779767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:13.779738 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:14.149724 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:28:14.149697 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd179689f_e41e_4d6d_b6b2_8aafe41e5c7b.slice/crio-176d3f4ebda7abaf8353391e1ea37aa36037d180ea52808e195f6ce0dafa8a37 WatchSource:0}: Error finding container 176d3f4ebda7abaf8353391e1ea37aa36037d180ea52808e195f6ce0dafa8a37: Status 404 returned error can't find the container with id 176d3f4ebda7abaf8353391e1ea37aa36037d180ea52808e195f6ce0dafa8a37 Nov 03 22:28:14.154216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:14.154192 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb"] Nov 03 22:28:14.582335 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:14.582295 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:14.912883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:14.912854 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" event={"ID":"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b","Type":"ContainerStarted","Data":"9a72c0f802eb01577342fde8a20783ae8f66465341b0f9c8e989909e77f5422d"} Nov 03 22:28:14.913045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:14.912907 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" event={"ID":"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b","Type":"ContainerStarted","Data":"176d3f4ebda7abaf8353391e1ea37aa36037d180ea52808e195f6ce0dafa8a37"} Nov 03 22:28:15.030760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:15.030707 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" podStartSLOduration=2.030693463 podStartE2EDuration="2.030693463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:15.027604097 +0000 UTC m=+1252.008337749" watchObservedRunningTime="2026-11-03 22:28:15.030693463 +0000 UTC m=+1252.011427113" Nov 03 22:28:15.920593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:15.920559 4298 generic.go:356] "Generic (PLEG): container finished" podID="d179689f-e41e-4d6d-b6b2-8aafe41e5c7b" containerID="9a72c0f802eb01577342fde8a20783ae8f66465341b0f9c8e989909e77f5422d" exitCode=0 Nov 03 22:28:15.920999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:15.920620 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" event={"ID":"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b","Type":"ContainerDied","Data":"9a72c0f802eb01577342fde8a20783ae8f66465341b0f9c8e989909e77f5422d"} Nov 03 22:28:17.255749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.255383 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:17.297153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.297115 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-xks-ca\") pod \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " Nov 03 22:28:17.297326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.297215 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-kube-api-access-fwpcm\" (UniqueName: \"kubernetes.io/projected/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-kube-api-access-fwpcm\") pod \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " Nov 03 22:28:17.297384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.297368 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-openshift-ca\") pod \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\" (UID: \"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b\") " Nov 03 22:28:17.297961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.297921 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-openshift-ca" pod "d179689f-e41e-4d6d-b6b2-8aafe41e5c7b" (UID: "d179689f-e41e-4d6d-b6b2-8aafe41e5c7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:17.299446 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.299421 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-xks-ca" pod "d179689f-e41e-4d6d-b6b2-8aafe41e5c7b" (UID: "d179689f-e41e-4d6d-b6b2-8aafe41e5c7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:17.299804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.299772 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-kube-api-access-fwpcm" pod "d179689f-e41e-4d6d-b6b2-8aafe41e5c7b" (UID: "d179689f-e41e-4d6d-b6b2-8aafe41e5c7b"). InnerVolumeSpecName "kube-api-access-fwpcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:17.401723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.401681 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:17.402172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.402150 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:17.402316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.402301 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwpcm\" (UniqueName: \"kubernetes.io/projected/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b-kube-api-access-fwpcm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:17.605752 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:28:17.605584 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:28:17.940427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.940344 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" Nov 03 22:28:17.940604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.940356 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb" event={"ID":"d179689f-e41e-4d6d-b6b2-8aafe41e5c7b","Type":"ContainerDied","Data":"176d3f4ebda7abaf8353391e1ea37aa36037d180ea52808e195f6ce0dafa8a37"} Nov 03 22:28:17.940604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:17.940473 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="176d3f4ebda7abaf8353391e1ea37aa36037d180ea52808e195f6ce0dafa8a37" Nov 03 22:28:19.046912 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:19.046876 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f"] Nov 03 22:28:19.098162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:19.098128 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c8eeaf-ncr4f"] Nov 03 22:28:19.136190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:19.136153 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf16452f-e7d7-4e81-b851-0c9269393e60" path="/var/lib/kubelet/pods/cf16452f-e7d7-4e81-b851-0c9269393e60/volumes" Nov 03 22:28:19.163944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:19.163909 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:19.407907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:19.407805 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:20.860996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:20.860961 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk"] Nov 03 22:28:20.861498 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:20.861484 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d179689f-e41e-4d6d-b6b2-8aafe41e5c7b" containerName="revoke-keys" Nov 03 22:28:20.861546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:20.861499 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d179689f-e41e-4d6d-b6b2-8aafe41e5c7b" containerName="revoke-keys" Nov 03 22:28:20.861610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:20.861600 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d179689f-e41e-4d6d-b6b2-8aafe41e5c7b" containerName="revoke-keys" Nov 03 22:28:20.865275 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:20.865256 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:20.917535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:20.917506 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk"] Nov 03 22:28:20.918377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:20.918351 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:20.918587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:20.918572 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:28:20.957451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:20.957419 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a292514-24bnk\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:20.957606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:20.957497 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a292514-24bnk\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:20.957732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:20.957711 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjx6z\" (UniqueName: \"kubernetes.io/projected/0107e812-54d9-4285-af97-7e41db1f17fc-kube-api-access-mjx6z\") pod \"maas-api-revoke-sub-5a292514-24bnk\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:21.059006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:21.058972 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a292514-24bnk\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:21.059148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:21.059076 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjx6z\" (UniqueName: \"kubernetes.io/projected/0107e812-54d9-4285-af97-7e41db1f17fc-kube-api-access-mjx6z\") pod \"maas-api-revoke-sub-5a292514-24bnk\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:21.059216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:21.059151 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a292514-24bnk\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:21.059606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:21.059584 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a292514-24bnk\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:21.061321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:21.061299 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a292514-24bnk\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:21.116726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:21.116662 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjx6z\" (UniqueName: \"kubernetes.io/projected/0107e812-54d9-4285-af97-7e41db1f17fc-kube-api-access-mjx6z\") pod \"maas-api-revoke-sub-5a292514-24bnk\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:21.178558 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:21.178528 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:21.549527 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:28:21.549488 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0107e812_54d9_4285_af97_7e41db1f17fc.slice/crio-baee083d288e72322142b5b5e532b77c95982c16022f7f9f87dc97d5820eb058 WatchSource:0}: Error finding container baee083d288e72322142b5b5e532b77c95982c16022f7f9f87dc97d5820eb058: Status 404 returned error can't find the container with id baee083d288e72322142b5b5e532b77c95982c16022f7f9f87dc97d5820eb058 Nov 03 22:28:21.552827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:21.552802 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk"] Nov 03 22:28:21.979493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:21.979461 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" event={"ID":"0107e812-54d9-4285-af97-7e41db1f17fc","Type":"ContainerStarted","Data":"2b9401eaa50a6aa93415e1cd47d72e29b3d4bf54e646be7d0aa0cc0ccd23feba"} Nov 03 22:28:21.979910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:21.979511 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" event={"ID":"0107e812-54d9-4285-af97-7e41db1f17fc","Type":"ContainerStarted","Data":"baee083d288e72322142b5b5e532b77c95982c16022f7f9f87dc97d5820eb058"} Nov 03 22:28:22.094135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:22.094041 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" podStartSLOduration=2.094029426 podStartE2EDuration="2.094029426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:22.093430658 +0000 UTC m=+1259.074164311" watchObservedRunningTime="2026-11-03 22:28:22.094029426 +0000 UTC m=+1259.074763079" Nov 03 22:28:22.988409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:22.988374 4298 generic.go:356] "Generic (PLEG): container finished" podID="0107e812-54d9-4285-af97-7e41db1f17fc" containerID="2b9401eaa50a6aa93415e1cd47d72e29b3d4bf54e646be7d0aa0cc0ccd23feba" exitCode=0 Nov 03 22:28:22.988864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:22.988427 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" event={"ID":"0107e812-54d9-4285-af97-7e41db1f17fc","Type":"ContainerDied","Data":"2b9401eaa50a6aa93415e1cd47d72e29b3d4bf54e646be7d0aa0cc0ccd23feba"} Nov 03 22:28:23.095991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:23.095954 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6"] Nov 03 22:28:23.098988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:23.098964 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3444cacd-kfnr6"] Nov 03 22:28:23.136028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:23.135994 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3970029b-1835-4668-be00-23fdd5511ce3" path="/var/lib/kubelet/pods/3970029b-1835-4668-be00-23fdd5511ce3/volumes" Nov 03 22:28:23.292338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:23.292245 4298 scope.go:122] "RemoveContainer" containerID="2e4c04b6db575ab8d0da83e52d1528b2d61a895168c988193137ba7dc3100e9a" Nov 03 22:28:23.304161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:23.304138 4298 scope.go:122] "RemoveContainer" containerID="7719e1a95dbc37a50233d229402d038da9170a7c60d5cc2ff823f48d910032d4" Nov 03 22:28:23.315619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:23.315593 4298 scope.go:122] "RemoveContainer" containerID="2e49c51d976ca66908da1f646c02a22a8fd4844d82c2a53800e93a9bce394e26" Nov 03 22:28:23.327226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:23.327201 4298 scope.go:122] "RemoveContainer" containerID="d81ce6e3725781af505cd07dda0127f6fefc95680f3ccf8bc99b84d5058f6a93" Nov 03 22:28:23.775469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:23.775434 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:24.155367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:24.155339 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:24.299071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:24.299034 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-xks-ca\") pod \"0107e812-54d9-4285-af97-7e41db1f17fc\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " Nov 03 22:28:24.299254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:24.299080 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0107e812-54d9-4285-af97-7e41db1f17fc-kube-api-access-mjx6z\" (UniqueName: \"kubernetes.io/projected/0107e812-54d9-4285-af97-7e41db1f17fc-kube-api-access-mjx6z\") pod \"0107e812-54d9-4285-af97-7e41db1f17fc\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " Nov 03 22:28:24.299399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:24.299371 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-openshift-ca\") pod \"0107e812-54d9-4285-af97-7e41db1f17fc\" (UID: \"0107e812-54d9-4285-af97-7e41db1f17fc\") " Nov 03 22:28:24.299865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:24.299800 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-openshift-ca" pod "0107e812-54d9-4285-af97-7e41db1f17fc" (UID: "0107e812-54d9-4285-af97-7e41db1f17fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:24.300029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:24.299999 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:24.301322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:24.301289 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-xks-ca" pod "0107e812-54d9-4285-af97-7e41db1f17fc" (UID: "0107e812-54d9-4285-af97-7e41db1f17fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:24.301436 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:24.301396 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0107e812-54d9-4285-af97-7e41db1f17fc-kube-api-access-mjx6z" pod "0107e812-54d9-4285-af97-7e41db1f17fc" (UID: "0107e812-54d9-4285-af97-7e41db1f17fc"). InnerVolumeSpecName "kube-api-access-mjx6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:24.401032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:24.400992 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0107e812-54d9-4285-af97-7e41db1f17fc-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:24.401032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:24.401026 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjx6z\" (UniqueName: \"kubernetes.io/projected/0107e812-54d9-4285-af97-7e41db1f17fc-kube-api-access-mjx6z\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:25.003157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:25.003122 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" event={"ID":"0107e812-54d9-4285-af97-7e41db1f17fc","Type":"ContainerDied","Data":"baee083d288e72322142b5b5e532b77c95982c16022f7f9f87dc97d5820eb058"} Nov 03 22:28:25.003345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:25.003167 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk" Nov 03 22:28:25.003345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:25.003173 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baee083d288e72322142b5b5e532b77c95982c16022f7f9f87dc97d5820eb058" Nov 03 22:28:27.515491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.515451 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd"] Nov 03 22:28:27.516005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.515990 4298 cpu_manager.go:414] "containerMap: removing container" podUID="0107e812-54d9-4285-af97-7e41db1f17fc" containerName="revoke-keys" Nov 03 22:28:27.516045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.516006 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0107e812-54d9-4285-af97-7e41db1f17fc" containerName="revoke-keys" Nov 03 22:28:27.516120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.516111 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="0107e812-54d9-4285-af97-7e41db1f17fc" containerName="revoke-keys" Nov 03 22:28:27.519425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.519409 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:27.573850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.573815 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd"] Nov 03 22:28:27.620075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.620041 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:27.620277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.620260 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:28:27.641512 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.641432 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf6da90f-z58pd\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:27.641512 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.641469 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxfx\" (UniqueName: \"kubernetes.io/projected/18f12da7-4753-4bee-ae76-74ac30f892f9-kube-api-access-ssxfx\") pod \"maas-api-revoke-sub-cf6da90f-z58pd\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:27.641512 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.641493 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf6da90f-z58pd\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:27.743114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.743082 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf6da90f-z58pd\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:27.743297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.743122 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxfx\" (UniqueName: \"kubernetes.io/projected/18f12da7-4753-4bee-ae76-74ac30f892f9-kube-api-access-ssxfx\") pod \"maas-api-revoke-sub-cf6da90f-z58pd\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:27.743391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.743371 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf6da90f-z58pd\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:27.744054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.744029 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf6da90f-z58pd\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:27.745498 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.745477 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf6da90f-z58pd\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:27.802007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.801973 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxfx\" (UniqueName: \"kubernetes.io/projected/18f12da7-4753-4bee-ae76-74ac30f892f9-kube-api-access-ssxfx\") pod \"maas-api-revoke-sub-cf6da90f-z58pd\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:27.832703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:27.832663 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:28.201133 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:28:28.201098 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f12da7_4753_4bee_ae76_74ac30f892f9.slice/crio-74fe669109551fc663c35f05256638fad1d1801d0d084dd79f2d2d780a3ce883 WatchSource:0}: Error finding container 74fe669109551fc663c35f05256638fad1d1801d0d084dd79f2d2d780a3ce883: Status 404 returned error can't find the container with id 74fe669109551fc663c35f05256638fad1d1801d0d084dd79f2d2d780a3ce883 Nov 03 22:28:28.204424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:28.204399 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd"] Nov 03 22:28:28.600512 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:28.600477 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:29.038469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:29.038432 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" event={"ID":"18f12da7-4753-4bee-ae76-74ac30f892f9","Type":"ContainerStarted","Data":"6d734eca6d88d51bf563d4c0d704a20f05545d6452fac4c18b9ab52f01c92b25"} Nov 03 22:28:29.038668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:29.038487 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" event={"ID":"18f12da7-4753-4bee-ae76-74ac30f892f9","Type":"ContainerStarted","Data":"74fe669109551fc663c35f05256638fad1d1801d0d084dd79f2d2d780a3ce883"} Nov 03 22:28:29.154238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:29.154188 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" podStartSLOduration=2.1541743 podStartE2EDuration="2.1541743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:29.153128629 +0000 UTC m=+1266.133862285" watchObservedRunningTime="2026-11-03 22:28:29.1541743 +0000 UTC m=+1266.134907973" Nov 03 22:28:30.046509 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:30.046472 4298 generic.go:356] "Generic (PLEG): container finished" podID="18f12da7-4753-4bee-ae76-74ac30f892f9" containerID="6d734eca6d88d51bf563d4c0d704a20f05545d6452fac4c18b9ab52f01c92b25" exitCode=0 Nov 03 22:28:30.046963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:30.046556 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" event={"ID":"18f12da7-4753-4bee-ae76-74ac30f892f9","Type":"ContainerDied","Data":"6d734eca6d88d51bf563d4c0d704a20f05545d6452fac4c18b9ab52f01c92b25"} Nov 03 22:28:31.210394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:31.210358 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:31.389554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:31.389465 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-openshift-ca\") pod \"18f12da7-4753-4bee-ae76-74ac30f892f9\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " Nov 03 22:28:31.389743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:31.389555 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-xks-ca\") pod \"18f12da7-4753-4bee-ae76-74ac30f892f9\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " Nov 03 22:28:31.389743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:31.389593 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18f12da7-4753-4bee-ae76-74ac30f892f9-kube-api-access-ssxfx\" (UniqueName: \"kubernetes.io/projected/18f12da7-4753-4bee-ae76-74ac30f892f9-kube-api-access-ssxfx\") pod \"18f12da7-4753-4bee-ae76-74ac30f892f9\" (UID: \"18f12da7-4753-4bee-ae76-74ac30f892f9\") " Nov 03 22:28:31.389972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:31.389947 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-openshift-ca" pod "18f12da7-4753-4bee-ae76-74ac30f892f9" (UID: "18f12da7-4753-4bee-ae76-74ac30f892f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:31.391731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:31.391708 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-xks-ca" pod "18f12da7-4753-4bee-ae76-74ac30f892f9" (UID: "18f12da7-4753-4bee-ae76-74ac30f892f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:31.391811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:31.391781 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f12da7-4753-4bee-ae76-74ac30f892f9-kube-api-access-ssxfx" pod "18f12da7-4753-4bee-ae76-74ac30f892f9" (UID: "18f12da7-4753-4bee-ae76-74ac30f892f9"). InnerVolumeSpecName "kube-api-access-ssxfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:31.490918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:31.490885 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:31.490918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:31.490912 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssxfx\" (UniqueName: \"kubernetes.io/projected/18f12da7-4753-4bee-ae76-74ac30f892f9-kube-api-access-ssxfx\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:31.490918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:31.490921 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18f12da7-4753-4bee-ae76-74ac30f892f9-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:32.061207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:32.061172 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" event={"ID":"18f12da7-4753-4bee-ae76-74ac30f892f9","Type":"ContainerDied","Data":"74fe669109551fc663c35f05256638fad1d1801d0d084dd79f2d2d780a3ce883"} Nov 03 22:28:32.061207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:32.061195 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd" Nov 03 22:28:32.061425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:32.061231 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74fe669109551fc663c35f05256638fad1d1801d0d084dd79f2d2d780a3ce883" Nov 03 22:28:33.179941 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:33.179907 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:34.055291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:34.055256 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x"] Nov 03 22:28:34.106887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:34.106850 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a70ecf-cmk4x"] Nov 03 22:28:35.049393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:35.049358 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7"] Nov 03 22:28:35.100248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:35.100218 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2386f0-qzfj7"] Nov 03 22:28:35.135366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:35.135335 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b049bc1-3d4a-4d78-ab23-b53dd95361d2" path="/var/lib/kubelet/pods/1b049bc1-3d4a-4d78-ab23-b53dd95361d2/volumes" Nov 03 22:28:35.135848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:35.135833 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e715dfd3-3388-4668-8795-699bb857671d" path="/var/lib/kubelet/pods/e715dfd3-3388-4668-8795-699bb857671d/volumes" Nov 03 22:28:37.687629 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:37.687588 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:40.770803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:40.770742 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:42.669238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.669199 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:42.786379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.786341 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x"] Nov 03 22:28:42.786976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.786955 4298 cpu_manager.go:414] "containerMap: removing container" podUID="18f12da7-4753-4bee-ae76-74ac30f892f9" containerName="revoke-keys" Nov 03 22:28:42.786976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.786974 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f12da7-4753-4bee-ae76-74ac30f892f9" containerName="revoke-keys" Nov 03 22:28:42.787141 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.787082 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f12da7-4753-4bee-ae76-74ac30f892f9" containerName="revoke-keys" Nov 03 22:28:42.790375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.790356 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.844285 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.844249 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x"] Nov 03 22:28:42.884983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.884950 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tmp-dir\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.884983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.884988 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-home\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.885238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.885133 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzwt\" (UniqueName: \"kubernetes.io/projected/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-kube-api-access-gmzwt\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.885238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.885182 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tls-certs\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.885362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.885348 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-dshm\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.885435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.885411 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-model-cache\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.891727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.891696 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"default-dockercfg-9jmx4\"" Nov 03 22:28:42.891727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.891706 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"openshift-service-ca.crt\"" Nov 03 22:28:42.891727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.891698 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"distinct-2-w1-3378fd-kserve-self-signed-certs\"" Nov 03 22:28:42.894315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.894300 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"kube-root-ca.crt\"" Nov 03 22:28:42.903760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.903736 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:42.987062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.987024 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzwt\" (UniqueName: \"kubernetes.io/projected/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-kube-api-access-gmzwt\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.987277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.987074 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tls-certs\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.987277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.987177 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-dshm\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.987277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.987215 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-model-cache\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.987448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.987327 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tmp-dir\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.987448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.987370 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-home\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.987802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.987766 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-model-cache\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.987802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.987784 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tmp-dir\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.988003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.987842 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-home\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.989635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.989611 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-dshm\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:42.989906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:42.989885 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tls-certs\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:43.048789 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:43.048754 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzwt\" (UniqueName: \"kubernetes.io/projected/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-kube-api-access-gmzwt\") pod \"distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:43.104298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:43.104259 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:43.540786 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:28:43.540750 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a2592c2_0c5e_4e4d_a01e_2fb1e382d3b0.slice/crio-1d6a344b144e4950202f1acd0882aa97e6e28504e386c710fef8c7a3c667b88e WatchSource:0}: Error finding container 1d6a344b144e4950202f1acd0882aa97e6e28504e386c710fef8c7a3c667b88e: Status 404 returned error can't find the container with id 1d6a344b144e4950202f1acd0882aa97e6e28504e386c710fef8c7a3c667b88e Nov 03 22:28:43.544387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:43.544362 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x"] Nov 03 22:28:44.176634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:44.176591 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" event={"ID":"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0","Type":"ContainerStarted","Data":"990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a"} Nov 03 22:28:44.177078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:44.176668 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" event={"ID":"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0","Type":"ContainerStarted","Data":"1d6a344b144e4950202f1acd0882aa97e6e28504e386c710fef8c7a3c667b88e"} Nov 03 22:28:44.294625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:44.294566 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podStartSLOduration=2.294551697 podStartE2EDuration="2.294551697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:44.293113839 +0000 UTC m=+1281.273847493" watchObservedRunningTime="2026-11-03 22:28:44.294551697 +0000 UTC m=+1281.275285351" Nov 03 22:28:44.866467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:44.866433 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct"] Nov 03 22:28:44.870687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:44.870669 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:44.910248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:44.910213 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c18b0ea2-gsdct\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:44.910457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:44.910436 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hd69\" (UniqueName: \"kubernetes.io/projected/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-kube-api-access-5hd69\") pod \"maas-api-revoke-sub-c18b0ea2-gsdct\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:44.910502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:44.910483 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c18b0ea2-gsdct\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:44.927274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:44.927241 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct"] Nov 03 22:28:44.973990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:44.973962 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:44.974158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:44.974015 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:28:45.011292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.011250 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hd69\" (UniqueName: \"kubernetes.io/projected/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-kube-api-access-5hd69\") pod \"maas-api-revoke-sub-c18b0ea2-gsdct\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:45.011478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.011316 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c18b0ea2-gsdct\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:45.011478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.011436 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c18b0ea2-gsdct\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:45.011973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.011953 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c18b0ea2-gsdct\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:45.013822 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.013792 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c18b0ea2-gsdct\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:45.048547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.048520 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm"] Nov 03 22:28:45.051741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.051718 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa1d926-hffqm"] Nov 03 22:28:45.069311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.069285 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hd69\" (UniqueName: \"kubernetes.io/projected/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-kube-api-access-5hd69\") pod \"maas-api-revoke-sub-c18b0ea2-gsdct\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:45.105369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.105335 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:45.135222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.135148 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021f11bd-d598-4905-b38f-55bb296889e8" path="/var/lib/kubelet/pods/021f11bd-d598-4905-b38f-55bb296889e8/volumes" Nov 03 22:28:45.184631 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.184598 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:45.553996 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:28:45.553965 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca3f8ad_3bf1_4aa9_9b87_093405c5536c.slice/crio-684367c0c6a6e3a9243e3b8f55e411947cb4bfd1b1d8f5afb6750e19019ef9c9 WatchSource:0}: Error finding container 684367c0c6a6e3a9243e3b8f55e411947cb4bfd1b1d8f5afb6750e19019ef9c9: Status 404 returned error can't find the container with id 684367c0c6a6e3a9243e3b8f55e411947cb4bfd1b1d8f5afb6750e19019ef9c9 Nov 03 22:28:45.557705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:45.557684 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct"] Nov 03 22:28:46.192010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:46.191917 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" event={"ID":"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c","Type":"ContainerStarted","Data":"45c4751860a810dc6a55ec205d7ec38f5f23da1603680a7c8f29d5cc37c13216"} Nov 03 22:28:46.192010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:46.191976 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" event={"ID":"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c","Type":"ContainerStarted","Data":"684367c0c6a6e3a9243e3b8f55e411947cb4bfd1b1d8f5afb6750e19019ef9c9"} Nov 03 22:28:46.308378 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:46.308313 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" podStartSLOduration=2.308297241 podStartE2EDuration="2.308297241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:46.305825155 +0000 UTC m=+1283.286558807" watchObservedRunningTime="2026-11-03 22:28:46.308297241 +0000 UTC m=+1283.289030894" Nov 03 22:28:47.199927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:47.199889 4298 generic.go:356] "Generic (PLEG): container finished" podID="8ca3f8ad-3bf1-4aa9-9b87-093405c5536c" containerID="45c4751860a810dc6a55ec205d7ec38f5f23da1603680a7c8f29d5cc37c13216" exitCode=0 Nov 03 22:28:47.200352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:47.199971 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" event={"ID":"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c","Type":"ContainerDied","Data":"45c4751860a810dc6a55ec205d7ec38f5f23da1603680a7c8f29d5cc37c13216"} Nov 03 22:28:47.433740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:47.433708 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:47.552980 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:28:47.552889 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:28:48.360185 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:48.360159 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:48.451584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:48.451551 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-openshift-ca\") pod \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " Nov 03 22:28:48.451768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:48.451681 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-xks-ca\") pod \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " Nov 03 22:28:48.451768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:48.451746 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-kube-api-access-5hd69\" (UniqueName: \"kubernetes.io/projected/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-kube-api-access-5hd69\") pod \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\" (UID: \"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c\") " Nov 03 22:28:48.452004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:48.451976 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-openshift-ca" pod "8ca3f8ad-3bf1-4aa9-9b87-093405c5536c" (UID: "8ca3f8ad-3bf1-4aa9-9b87-093405c5536c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:48.452218 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:48.452200 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:48.453849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:48.453826 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-kube-api-access-5hd69" pod "8ca3f8ad-3bf1-4aa9-9b87-093405c5536c" (UID: "8ca3f8ad-3bf1-4aa9-9b87-093405c5536c"). InnerVolumeSpecName "kube-api-access-5hd69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:48.453938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:48.453825 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-xks-ca" pod "8ca3f8ad-3bf1-4aa9-9b87-093405c5536c" (UID: "8ca3f8ad-3bf1-4aa9-9b87-093405c5536c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:48.553189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:48.553155 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:48.553189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:48.553185 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hd69\" (UniqueName: \"kubernetes.io/projected/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c-kube-api-access-5hd69\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:49.213554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:49.213524 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" Nov 03 22:28:49.213747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:49.213559 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct" event={"ID":"8ca3f8ad-3bf1-4aa9-9b87-093405c5536c","Type":"ContainerDied","Data":"684367c0c6a6e3a9243e3b8f55e411947cb4bfd1b1d8f5afb6750e19019ef9c9"} Nov 03 22:28:49.213747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:49.213607 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="684367c0c6a6e3a9243e3b8f55e411947cb4bfd1b1d8f5afb6750e19019ef9c9" Nov 03 22:28:50.583101 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.582930 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg"] Nov 03 22:28:50.583499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.583448 4298 cpu_manager.go:414] "containerMap: removing container" podUID="8ca3f8ad-3bf1-4aa9-9b87-093405c5536c" containerName="revoke-keys" Nov 03 22:28:50.583499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.583461 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca3f8ad-3bf1-4aa9-9b87-093405c5536c" containerName="revoke-keys" Nov 03 22:28:50.583580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.583569 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca3f8ad-3bf1-4aa9-9b87-093405c5536c" containerName="revoke-keys" Nov 03 22:28:50.587031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.587013 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:50.640849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.640817 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg"] Nov 03 22:28:50.678933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.678900 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7cw\" (UniqueName: \"kubernetes.io/projected/66e58db0-fee5-4007-a5ba-0c7f181d7f08-kube-api-access-7w7cw\") pod \"maas-api-revoke-sub-6a5d667a-7hvlg\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:50.679118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.679078 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5d667a-7hvlg\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:50.679172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.679131 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5d667a-7hvlg\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:50.686948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.686923 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:50.687072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.687034 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:28:50.780600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.780565 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7cw\" (UniqueName: \"kubernetes.io/projected/66e58db0-fee5-4007-a5ba-0c7f181d7f08-kube-api-access-7w7cw\") pod \"maas-api-revoke-sub-6a5d667a-7hvlg\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:50.780805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.780696 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5d667a-7hvlg\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:50.780805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.780733 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5d667a-7hvlg\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:50.781360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.781337 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5d667a-7hvlg\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:50.782962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.782945 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5d667a-7hvlg\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:50.839125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.839052 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7cw\" (UniqueName: \"kubernetes.io/projected/66e58db0-fee5-4007-a5ba-0c7f181d7f08-kube-api-access-7w7cw\") pod \"maas-api-revoke-sub-6a5d667a-7hvlg\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:50.900327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:50.900296 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:51.049528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:51.049496 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld"] Nov 03 22:28:51.053974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:51.053950 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1adc586f-4hhld"] Nov 03 22:28:51.056711 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:51.056696 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk"] Nov 03 22:28:51.059176 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:51.059155 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39feb93e-44pqk"] Nov 03 22:28:51.136038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:51.135950 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="397ed36d-1d6d-47da-bde1-fca932d0951a" path="/var/lib/kubelet/pods/397ed36d-1d6d-47da-bde1-fca932d0951a/volumes" Nov 03 22:28:51.136528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:51.136505 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b9c655-b581-4633-9030-4a5590464160" path="/var/lib/kubelet/pods/d0b9c655-b581-4633-9030-4a5590464160/volumes" Nov 03 22:28:51.272161 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:28:51.272131 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e58db0_fee5_4007_a5ba_0c7f181d7f08.slice/crio-526e6f29f8fc213402d7cec66763298230bf07e054b77df64ac38bcb2e297f0c WatchSource:0}: Error finding container 526e6f29f8fc213402d7cec66763298230bf07e054b77df64ac38bcb2e297f0c: Status 404 returned error can't find the container with id 526e6f29f8fc213402d7cec66763298230bf07e054b77df64ac38bcb2e297f0c Nov 03 22:28:51.276574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:51.276550 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg"] Nov 03 22:28:51.888868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:51.888780 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:52.234714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:52.234677 4298 generic.go:356] "Generic (PLEG): container finished" podID="66e58db0-fee5-4007-a5ba-0c7f181d7f08" containerID="5064c4b231a91e48980dd8c760ed4f14e239d8bf0d1528f9f62923611d5df3ff" exitCode=0 Nov 03 22:28:52.234893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:52.234768 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" event={"ID":"66e58db0-fee5-4007-a5ba-0c7f181d7f08","Type":"ContainerDied","Data":"5064c4b231a91e48980dd8c760ed4f14e239d8bf0d1528f9f62923611d5df3ff"} Nov 03 22:28:52.234893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:52.234819 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" event={"ID":"66e58db0-fee5-4007-a5ba-0c7f181d7f08","Type":"ContainerStarted","Data":"526e6f29f8fc213402d7cec66763298230bf07e054b77df64ac38bcb2e297f0c"} Nov 03 22:28:53.104889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.104853 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:53.118751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.118724 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:53.253710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.253683 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:28:53.393422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.393397 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:53.515607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.515577 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e58db0-fee5-4007-a5ba-0c7f181d7f08-kube-api-access-7w7cw\" (UniqueName: \"kubernetes.io/projected/66e58db0-fee5-4007-a5ba-0c7f181d7f08-kube-api-access-7w7cw\") pod \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " Nov 03 22:28:53.515810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.515614 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-xks-ca\") pod \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " Nov 03 22:28:53.515810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.515777 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-openshift-ca\") pod \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\" (UID: \"66e58db0-fee5-4007-a5ba-0c7f181d7f08\") " Nov 03 22:28:53.516187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.516160 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-openshift-ca" pod "66e58db0-fee5-4007-a5ba-0c7f181d7f08" (UID: "66e58db0-fee5-4007-a5ba-0c7f181d7f08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:53.516423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.516403 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:53.518016 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.517989 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-xks-ca" pod "66e58db0-fee5-4007-a5ba-0c7f181d7f08" (UID: "66e58db0-fee5-4007-a5ba-0c7f181d7f08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:53.518135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.518012 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e58db0-fee5-4007-a5ba-0c7f181d7f08-kube-api-access-7w7cw" pod "66e58db0-fee5-4007-a5ba-0c7f181d7f08" (UID: "66e58db0-fee5-4007-a5ba-0c7f181d7f08"). InnerVolumeSpecName "kube-api-access-7w7cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:53.617609 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.617525 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w7cw\" (UniqueName: \"kubernetes.io/projected/66e58db0-fee5-4007-a5ba-0c7f181d7f08-kube-api-access-7w7cw\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:53.617609 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:53.617552 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e58db0-fee5-4007-a5ba-0c7f181d7f08-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:54.251126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:54.251097 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" Nov 03 22:28:54.251618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:54.251138 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg" event={"ID":"66e58db0-fee5-4007-a5ba-0c7f181d7f08","Type":"ContainerDied","Data":"526e6f29f8fc213402d7cec66763298230bf07e054b77df64ac38bcb2e297f0c"} Nov 03 22:28:54.251618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:54.251189 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="526e6f29f8fc213402d7cec66763298230bf07e054b77df64ac38bcb2e297f0c" Nov 03 22:28:55.593780 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.593746 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x"] Nov 03 22:28:55.594291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.594274 4298 cpu_manager.go:414] "containerMap: removing container" podUID="66e58db0-fee5-4007-a5ba-0c7f181d7f08" containerName="revoke-keys" Nov 03 22:28:55.594291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.594290 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e58db0-fee5-4007-a5ba-0c7f181d7f08" containerName="revoke-keys" Nov 03 22:28:55.594402 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.594393 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e58db0-fee5-4007-a5ba-0c7f181d7f08" containerName="revoke-keys" Nov 03 22:28:55.597960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.597944 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:55.655530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.655494 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x"] Nov 03 22:28:55.702654 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.702620 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:55.702836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.702672 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:28:55.745614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.745581 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c933eb6-6d64x\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:55.745830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.745698 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c933eb6-6d64x\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:55.745830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.745814 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5l4\" (UniqueName: \"kubernetes.io/projected/ced587dd-6d32-40c2-816e-e6c9208d356f-kube-api-access-fn5l4\") pod \"maas-api-revoke-sub-4c933eb6-6d64x\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:55.846945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.846859 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c933eb6-6d64x\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:55.847088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.846951 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c933eb6-6d64x\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:55.847088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.847049 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5l4\" (UniqueName: \"kubernetes.io/projected/ced587dd-6d32-40c2-816e-e6c9208d356f-kube-api-access-fn5l4\") pod \"maas-api-revoke-sub-4c933eb6-6d64x\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:55.847725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.847673 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c933eb6-6d64x\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:55.849334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.849310 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c933eb6-6d64x\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:55.906405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.906366 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5l4\" (UniqueName: \"kubernetes.io/projected/ced587dd-6d32-40c2-816e-e6c9208d356f-kube-api-access-fn5l4\") pod \"maas-api-revoke-sub-4c933eb6-6d64x\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:55.915218 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:55.915192 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:56.286281 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:28:56.286250 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced587dd_6d32_40c2_816e_e6c9208d356f.slice/crio-1a635572b13f4877aba96f67096adfffd94af064295120c48e4f7872315a3e85 WatchSource:0}: Error finding container 1a635572b13f4877aba96f67096adfffd94af064295120c48e4f7872315a3e85: Status 404 returned error can't find the container with id 1a635572b13f4877aba96f67096adfffd94af064295120c48e4f7872315a3e85 Nov 03 22:28:56.289245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:56.289214 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x"] Nov 03 22:28:56.509364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:56.509337 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:57.270870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:57.270833 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" event={"ID":"ced587dd-6d32-40c2-816e-e6c9208d356f","Type":"ContainerStarted","Data":"cb8400922a0555c4c552707a00a3402f7b06d84bcd209d94ce273e81405d38f3"} Nov 03 22:28:57.271257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:57.270888 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" event={"ID":"ced587dd-6d32-40c2-816e-e6c9208d356f","Type":"ContainerStarted","Data":"1a635572b13f4877aba96f67096adfffd94af064295120c48e4f7872315a3e85"} Nov 03 22:28:57.462431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:57.462376 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" podStartSLOduration=2.462361377 podStartE2EDuration="2.462361377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:57.457804304 +0000 UTC m=+1294.438537957" watchObservedRunningTime="2026-11-03 22:28:57.462361377 +0000 UTC m=+1294.443095023" Nov 03 22:28:57.463111 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:57.463087 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7"] Nov 03 22:28:57.463307 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:57.463284 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" containerID="cri-o://a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc" gracePeriod=30 Nov 03 22:28:58.280570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:58.280531 4298 generic.go:356] "Generic (PLEG): container finished" podID="ced587dd-6d32-40c2-816e-e6c9208d356f" containerID="cb8400922a0555c4c552707a00a3402f7b06d84bcd209d94ce273e81405d38f3" exitCode=0 Nov 03 22:28:58.280994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:58.280614 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" event={"ID":"ced587dd-6d32-40c2-816e-e6c9208d356f","Type":"ContainerDied","Data":"cb8400922a0555c4c552707a00a3402f7b06d84bcd209d94ce273e81405d38f3"} Nov 03 22:28:59.049397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.049357 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg"] Nov 03 22:28:59.100589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.100553 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890b359d-lrtzg"] Nov 03 22:28:59.134574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.134531 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a78a206a-9a9d-4286-a4a4-b12929c75977" path="/var/lib/kubelet/pods/a78a206a-9a9d-4286-a4a4-b12929c75977/volumes" Nov 03 22:28:59.442399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.442373 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:28:59.595237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.595158 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced587dd-6d32-40c2-816e-e6c9208d356f-kube-api-access-fn5l4\" (UniqueName: \"kubernetes.io/projected/ced587dd-6d32-40c2-816e-e6c9208d356f-kube-api-access-fn5l4\") pod \"ced587dd-6d32-40c2-816e-e6c9208d356f\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " Nov 03 22:28:59.595237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.595205 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-xks-ca\") pod \"ced587dd-6d32-40c2-816e-e6c9208d356f\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " Nov 03 22:28:59.595237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.595226 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-openshift-ca\") pod \"ced587dd-6d32-40c2-816e-e6c9208d356f\" (UID: \"ced587dd-6d32-40c2-816e-e6c9208d356f\") " Nov 03 22:28:59.595692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.595663 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-openshift-ca" pod "ced587dd-6d32-40c2-816e-e6c9208d356f" (UID: "ced587dd-6d32-40c2-816e-e6c9208d356f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:59.595860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.595840 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:59.597299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.597271 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-xks-ca" pod "ced587dd-6d32-40c2-816e-e6c9208d356f" (UID: "ced587dd-6d32-40c2-816e-e6c9208d356f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:59.597416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.597390 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced587dd-6d32-40c2-816e-e6c9208d356f-kube-api-access-fn5l4" pod "ced587dd-6d32-40c2-816e-e6c9208d356f" (UID: "ced587dd-6d32-40c2-816e-e6c9208d356f"). InnerVolumeSpecName "kube-api-access-fn5l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:59.697206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.697163 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5l4\" (UniqueName: \"kubernetes.io/projected/ced587dd-6d32-40c2-816e-e6c9208d356f-kube-api-access-fn5l4\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:59.697386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:28:59.697228 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced587dd-6d32-40c2-816e-e6c9208d356f-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:00.294926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:00.294889 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" event={"ID":"ced587dd-6d32-40c2-816e-e6c9208d356f","Type":"ContainerDied","Data":"1a635572b13f4877aba96f67096adfffd94af064295120c48e4f7872315a3e85"} Nov 03 22:29:00.295072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:00.294943 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a635572b13f4877aba96f67096adfffd94af064295120c48e4f7872315a3e85" Nov 03 22:29:00.295072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:00.294905 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x" Nov 03 22:29:01.109713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:01.109670 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:02.738911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.738839 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:29:02.841085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841046 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-socket\") pod \"317f4b1e-43be-46a0-861e-f72df5c135ce\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " Nov 03 22:29:02.841263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841139 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-token\" (UniqueName: \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-token\") pod \"317f4b1e-43be-46a0-861e-f72df5c135ce\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " Nov 03 22:29:02.841263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841158 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-credential-socket\") pod \"317f4b1e-43be-46a0-861e-f72df5c135ce\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " Nov 03 22:29:02.841263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841214 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-kube-api-access-9ww9p\" (UniqueName: \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-kube-api-access-9ww9p\") pod \"317f4b1e-43be-46a0-861e-f72df5c135ce\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " Nov 03 22:29:02.841263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841254 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-envoy\") pod \"317f4b1e-43be-46a0-861e-f72df5c135ce\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " Nov 03 22:29:02.841430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841334 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-certs\") pod \"317f4b1e-43be-46a0-861e-f72df5c135ce\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " Nov 03 22:29:02.841430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841351 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-socket" pod "317f4b1e-43be-46a0-861e-f72df5c135ce" (UID: "317f4b1e-43be-46a0-861e-f72df5c135ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:29:02.841430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841361 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317f4b1e-43be-46a0-861e-f72df5c135ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317f4b1e-43be-46a0-861e-f72df5c135ce-istiod-ca-cert\") pod \"317f4b1e-43be-46a0-861e-f72df5c135ce\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " Nov 03 22:29:02.841565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841477 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-data\") pod \"317f4b1e-43be-46a0-861e-f72df5c135ce\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " Nov 03 22:29:02.841565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841488 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-credential-socket" pod "317f4b1e-43be-46a0-861e-f72df5c135ce" (UID: "317f4b1e-43be-46a0-861e-f72df5c135ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:29:02.841696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841563 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-podinfo\") pod \"317f4b1e-43be-46a0-861e-f72df5c135ce\" (UID: \"317f4b1e-43be-46a0-861e-f72df5c135ce\") " Nov 03 22:29:02.841696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841685 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-certs" pod "317f4b1e-43be-46a0-861e-f72df5c135ce" (UID: "317f4b1e-43be-46a0-861e-f72df5c135ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:29:02.841813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841713 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-data" pod "317f4b1e-43be-46a0-861e-f72df5c135ce" (UID: "317f4b1e-43be-46a0-861e-f72df5c135ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:29:02.841813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.841764 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317f4b1e-43be-46a0-861e-f72df5c135ce-istiod-ca-cert" pod "317f4b1e-43be-46a0-861e-f72df5c135ce" (UID: "317f4b1e-43be-46a0-861e-f72df5c135ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:02.842182 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.842162 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:02.842303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.842185 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317f4b1e-43be-46a0-861e-f72df5c135ce-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:02.842303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.842200 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:02.842303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.842214 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:02.842303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.842228 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:02.844030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.844001 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-token" pod "317f4b1e-43be-46a0-861e-f72df5c135ce" (UID: "317f4b1e-43be-46a0-861e-f72df5c135ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:02.844152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.844028 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-kube-api-access-9ww9p" pod "317f4b1e-43be-46a0-861e-f72df5c135ce" (UID: "317f4b1e-43be-46a0-861e-f72df5c135ce"). InnerVolumeSpecName "kube-api-access-9ww9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:02.844201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.844149 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-podinfo" pod "317f4b1e-43be-46a0-861e-f72df5c135ce" (UID: "317f4b1e-43be-46a0-861e-f72df5c135ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:29:02.844201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.844160 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-envoy" pod "317f4b1e-43be-46a0-861e-f72df5c135ce" (UID: "317f4b1e-43be-46a0-861e-f72df5c135ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:29:02.943133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.943042 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:02.943133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.943079 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ww9p\" (UniqueName: \"kubernetes.io/projected/317f4b1e-43be-46a0-861e-f72df5c135ce-kube-api-access-9ww9p\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:02.943133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.943090 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:02.943133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:02.943101 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317f4b1e-43be-46a0-861e-f72df5c135ce-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:03.315841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.315797 4298 generic.go:356] "Generic (PLEG): container finished" podID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerID="a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc" exitCode=0 Nov 03 22:29:03.315996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.315874 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" Nov 03 22:29:03.315996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.315880 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" event={"ID":"317f4b1e-43be-46a0-861e-f72df5c135ce","Type":"ContainerDied","Data":"a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc"} Nov 03 22:29:03.315996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.315938 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7" event={"ID":"317f4b1e-43be-46a0-861e-f72df5c135ce","Type":"ContainerDied","Data":"ff3bff6cb0f4e7cbd685c7cf13978c5b5c2cb06cef2dff24f6dd54190c12e969"} Nov 03 22:29:03.315996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.315961 4298 scope.go:122] "RemoveContainer" containerID="a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc" Nov 03 22:29:03.325989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.325967 4298 scope.go:122] "RemoveContainer" containerID="ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb" Nov 03 22:29:03.335634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.335609 4298 scope.go:122] "RemoveContainer" containerID="a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc" Nov 03 22:29:03.335923 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:29:03.335905 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc\": container with ID starting with a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc not found: ID does not exist" containerID="a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc" Nov 03 22:29:03.335984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.335932 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc"} err="failed to get container status \"a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc\": rpc error: code = NotFound desc = could not find container \"a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc\": container with ID starting with a8fe6ba14bd31c1a8f4a10b931e710d55607830997057339e0a9e891a45f4ebc not found: ID does not exist" Nov 03 22:29:03.335984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.335951 4298 scope.go:122] "RemoveContainer" containerID="ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb" Nov 03 22:29:03.336242 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:29:03.336214 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb\": container with ID starting with ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb not found: ID does not exist" containerID="ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb" Nov 03 22:29:03.336286 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.336249 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb"} err="failed to get container status \"ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb\": rpc error: code = NotFound desc = could not find container \"ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb\": container with ID starting with ea5c0b126c50a23077aa832aedc1d1842c5fa6b2557c150e899e97689128cdfb not found: ID does not exist" Nov 03 22:29:03.486150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.486115 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7"] Nov 03 22:29:03.489929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:03.489898 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-jc9l7"] Nov 03 22:29:05.135496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:05.135463 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" path="/var/lib/kubelet/pods/317f4b1e-43be-46a0-861e-f72df5c135ce/volumes" Nov 03 22:29:05.800527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:05.800497 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:06.048971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.048941 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7"] Nov 03 22:29:06.100123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.100044 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93848d8-bp9k7"] Nov 03 22:29:06.102783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.102763 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l"] Nov 03 22:29:06.103241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.103230 4298 cpu_manager.go:414] "containerMap: removing container" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" Nov 03 22:29:06.103283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.103243 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" Nov 03 22:29:06.103317 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.103285 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ced587dd-6d32-40c2-816e-e6c9208d356f" containerName="revoke-keys" Nov 03 22:29:06.103317 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.103290 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced587dd-6d32-40c2-816e-e6c9208d356f" containerName="revoke-keys" Nov 03 22:29:06.103387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.103363 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" Nov 03 22:29:06.103387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.103370 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced587dd-6d32-40c2-816e-e6c9208d356f" containerName="revoke-keys" Nov 03 22:29:06.103387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.103377 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" Nov 03 22:29:06.108466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.108451 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:06.153249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.153220 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l"] Nov 03 22:29:06.161865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.161837 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:06.162023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.161905 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:06.179628 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.179598 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9ce596-lvp6l\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:06.179809 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.179746 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9ce596-lvp6l\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:06.179860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.179845 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5zqb\" (UniqueName: \"kubernetes.io/projected/044db9d5-efe9-4592-b923-4c86c8203a2a-kube-api-access-s5zqb\") pod \"maas-api-revoke-sub-6e9ce596-lvp6l\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:06.280953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.280921 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9ce596-lvp6l\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:06.281158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.280997 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9ce596-lvp6l\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:06.281158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.281054 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5zqb\" (UniqueName: \"kubernetes.io/projected/044db9d5-efe9-4592-b923-4c86c8203a2a-kube-api-access-s5zqb\") pod \"maas-api-revoke-sub-6e9ce596-lvp6l\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:06.281769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.281739 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9ce596-lvp6l\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:06.283388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.283366 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9ce596-lvp6l\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:06.339164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.339129 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5zqb\" (UniqueName: \"kubernetes.io/projected/044db9d5-efe9-4592-b923-4c86c8203a2a-kube-api-access-s5zqb\") pod \"maas-api-revoke-sub-6e9ce596-lvp6l\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:06.418475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.418387 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8"] Nov 03 22:29:06.419088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.419066 4298 cpu_manager.go:414] "containerMap: removing container" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" Nov 03 22:29:06.419159 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.419091 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317f4b1e-43be-46a0-861e-f72df5c135ce" containerName="istio-proxy" Nov 03 22:29:06.421794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.421774 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:06.423161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.423144 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:06.466228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.466194 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8"] Nov 03 22:29:06.486829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.486015 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b8489bb-s57n8\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:06.486829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.486063 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdbm6\" (UniqueName: \"kubernetes.io/projected/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-kube-api-access-hdbm6\") pod \"maas-api-revoke-sub-1b8489bb-s57n8\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:06.486829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.486119 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b8489bb-s57n8\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:06.587830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.587794 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b8489bb-s57n8\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:06.587830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.587833 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdbm6\" (UniqueName: \"kubernetes.io/projected/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-kube-api-access-hdbm6\") pod \"maas-api-revoke-sub-1b8489bb-s57n8\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:06.588099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.587877 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b8489bb-s57n8\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:06.588527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.588500 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b8489bb-s57n8\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:06.590325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.590299 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b8489bb-s57n8\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:06.646722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.646690 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdbm6\" (UniqueName: \"kubernetes.io/projected/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-kube-api-access-hdbm6\") pod \"maas-api-revoke-sub-1b8489bb-s57n8\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:06.763746 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.763681 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:06.781934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.781908 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn"] Nov 03 22:29:06.786130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.786109 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.802966 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:29:06.802938 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044db9d5_efe9_4592_b923_4c86c8203a2a.slice/crio-32bb8fef3f737ec7f750e832fca941ba1635deedf7f1053efbc94a72c0ff0f15 WatchSource:0}: Error finding container 32bb8fef3f737ec7f750e832fca941ba1635deedf7f1053efbc94a72c0ff0f15: Status 404 returned error can't find the container with id 32bb8fef3f737ec7f750e832fca941ba1635deedf7f1053efbc94a72c0ff0f15 Nov 03 22:29:06.805716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.805690 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l"] Nov 03 22:29:06.864123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.857809 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn"] Nov 03 22:29:06.891222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.891186 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c87f27-faa4-4ce3-8c24-939548b06801-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.891408 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.891231 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.891408 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.891340 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.891519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.891439 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.891574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.891515 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.891574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.891539 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.891710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.891631 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.891840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.891818 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b58s\" (UniqueName: \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-kube-api-access-7b58s\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.891908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.891853 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993078 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993130 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993146 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993263 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993394 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b58s\" (UniqueName: \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-kube-api-access-7b58s\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993417 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993470 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c87f27-faa4-4ce3-8c24-939548b06801-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993516 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993558 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993574 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993587 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993855 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.993944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.993921 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.994232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.994212 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c87f27-faa4-4ce3-8c24-939548b06801-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.995595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.995577 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:06.995841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:06.995815 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:07.054225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.054194 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:07.056697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.056677 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b58s\" (UniqueName: \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-kube-api-access-7b58s\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:07.101539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.101505 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:07.101742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.101685 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz"] Nov 03 22:29:07.106362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.106348 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.135705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.135666 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb954196-4b0b-4777-bb5b-ecb8fd5328a1" path="/var/lib/kubelet/pods/cb954196-4b0b-4777-bb5b-ecb8fd5328a1/volumes" Nov 03 22:29:07.151813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.151789 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8"] Nov 03 22:29:07.169835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.169806 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz"] Nov 03 22:29:07.196808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.196250 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.196808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.196298 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.196808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.196341 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.196808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.196379 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.196808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.196591 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.196808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.196630 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.196808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.196767 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dck7c\" (UniqueName: \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-kube-api-access-dck7c\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.196808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.196803 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.197150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.196879 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.297938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.297852 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.297938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.297893 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.297966 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.297983 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.298029 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dck7c\" (UniqueName: \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-kube-api-access-dck7c\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.298059 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.298112 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.298168 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.298195 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.298672 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.298799 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.298814 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.298861 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.298967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.298893 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.324499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.324457 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.324818 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.324796 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.347323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.347203 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" event={"ID":"044db9d5-efe9-4592-b923-4c86c8203a2a","Type":"ContainerStarted","Data":"eb0c251bad35e933d5585619e3b0acc1231a0b01c782156fb8e085b63ea396d5"} Nov 03 22:29:07.347323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.347300 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" event={"ID":"044db9d5-efe9-4592-b923-4c86c8203a2a","Type":"ContainerStarted","Data":"32bb8fef3f737ec7f750e832fca941ba1635deedf7f1053efbc94a72c0ff0f15"} Nov 03 22:29:07.348795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.348767 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" event={"ID":"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c","Type":"ContainerStarted","Data":"143b6b08399b259357290e20516a83a4fe766a53fb5227efde0eabfb3fadc85b"} Nov 03 22:29:07.356274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.356243 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.356568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.356545 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dck7c\" (UniqueName: \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-kube-api-access-dck7c\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.419990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.419958 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:07.462597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.462544 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" podStartSLOduration=1.462528778 podStartE2EDuration="1.462528778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:07.46210466 +0000 UTC m=+1304.442838313" watchObservedRunningTime="2026-11-03 22:29:07.462528778 +0000 UTC m=+1304.443262431" Nov 03 22:29:07.494978 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:29:07.494945 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c87f27_faa4_4ce3_8c24_939548b06801.slice/crio-76fa50a592fbb46132b4dca8be195b83cd7518f08f311adbd4cfac62d6e33828 WatchSource:0}: Error finding container 76fa50a592fbb46132b4dca8be195b83cd7518f08f311adbd4cfac62d6e33828: Status 404 returned error can't find the container with id 76fa50a592fbb46132b4dca8be195b83cd7518f08f311adbd4cfac62d6e33828 Nov 03 22:29:07.500187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.500145 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:07.500305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.500232 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:07.500305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.500275 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:07.501810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.501787 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn"] Nov 03 22:29:07.801962 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:29:07.801935 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cfec8a_343e_42fc_bf6e_8dcc495c1220.slice/crio-6d3cd5198d4209bf721eeb0d935dc37e1e57d0088027d5b39c6925070b40cb17 WatchSource:0}: Error finding container 6d3cd5198d4209bf721eeb0d935dc37e1e57d0088027d5b39c6925070b40cb17: Status 404 returned error can't find the container with id 6d3cd5198d4209bf721eeb0d935dc37e1e57d0088027d5b39c6925070b40cb17 Nov 03 22:29:07.803460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.803429 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz"] Nov 03 22:29:07.805127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.805089 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:07.805203 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.805173 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:07.805241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:07.805217 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:08.156545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.156448 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt"] Nov 03 22:29:08.160623 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.160601 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.209308 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.209273 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.209731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.209328 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.209731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.209471 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.209731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.209503 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp7sb\" (UniqueName: \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-kube-api-access-lp7sb\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.209731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.209565 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.209942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.209848 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.209942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.209893 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debadcac-80e7-4843-a84a-dc60f364566c-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.210174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.210152 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debadcac-80e7-4843-a84a-dc60f364566c-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.210332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.210308 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.264555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.264520 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt"] Nov 03 22:29:08.311785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.311751 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.311967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.311854 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.311967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.311887 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.311967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.311949 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.312137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.311976 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7sb\" (UniqueName: \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-kube-api-access-lp7sb\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.312137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.312002 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.312137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.312072 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.312137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.312109 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debadcac-80e7-4843-a84a-dc60f364566c-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.312343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.312155 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.312343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.312179 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debadcac-80e7-4843-a84a-dc60f364566c-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.312343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.312323 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.312496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.312421 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.312584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.312559 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.312968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.312926 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debadcac-80e7-4843-a84a-dc60f364566c-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.314281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.314262 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.314577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.314558 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debadcac-80e7-4843-a84a-dc60f364566c-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.363804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.363766 4298 generic.go:356] "Generic (PLEG): container finished" podID="044db9d5-efe9-4592-b923-4c86c8203a2a" containerID="eb0c251bad35e933d5585619e3b0acc1231a0b01c782156fb8e085b63ea396d5" exitCode=0 Nov 03 22:29:08.364003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.363855 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" event={"ID":"044db9d5-efe9-4592-b923-4c86c8203a2a","Type":"ContainerDied","Data":"eb0c251bad35e933d5585619e3b0acc1231a0b01c782156fb8e085b63ea396d5"} Nov 03 22:29:08.365670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.365613 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" event={"ID":"d8c87f27-faa4-4ce3-8c24-939548b06801","Type":"ContainerStarted","Data":"de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7"} Nov 03 22:29:08.365803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.365702 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" event={"ID":"d8c87f27-faa4-4ce3-8c24-939548b06801","Type":"ContainerStarted","Data":"76fa50a592fbb46132b4dca8be195b83cd7518f08f311adbd4cfac62d6e33828"} Nov 03 22:29:08.368355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.368325 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" event={"ID":"43cfec8a-343e-42fc-bf6e-8dcc495c1220","Type":"ContainerStarted","Data":"b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918"} Nov 03 22:29:08.368485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.368373 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" event={"ID":"43cfec8a-343e-42fc-bf6e-8dcc495c1220","Type":"ContainerStarted","Data":"6d3cd5198d4209bf721eeb0d935dc37e1e57d0088027d5b39c6925070b40cb17"} Nov 03 22:29:08.370186 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.370161 4298 generic.go:356] "Generic (PLEG): container finished" podID="0234c2e8-ecd4-4b8e-a523-4cab2501ad3c" containerID="4abe882594d9cd339a20ed561aeb712e180cb921d7f425be2131788eaa4abfdb" exitCode=0 Nov 03 22:29:08.370291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.370254 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" event={"ID":"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c","Type":"ContainerDied","Data":"4abe882594d9cd339a20ed561aeb712e180cb921d7f425be2131788eaa4abfdb"} Nov 03 22:29:08.371213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.371186 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.371477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.371456 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp7sb\" (UniqueName: \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-kube-api-access-lp7sb\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.421177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.421102 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:08.422892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.422862 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:29:08.423038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.422917 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:29:08.478700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.475991 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:08.684136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.684008 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podStartSLOduration=2.683987106 podStartE2EDuration="2.683987106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:08.682974497 +0000 UTC m=+1305.663708151" watchObservedRunningTime="2026-11-03 22:29:08.683987106 +0000 UTC m=+1305.664720765" Nov 03 22:29:08.861665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.852956 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podStartSLOduration=1.852933374 podStartE2EDuration="1.852933374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:08.824008052 +0000 UTC m=+1305.804741706" watchObservedRunningTime="2026-11-03 22:29:08.852933374 +0000 UTC m=+1305.833667027" Nov 03 22:29:08.968714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:08.953736 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt"] Nov 03 22:29:09.005056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.004634 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:09.005056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.004747 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:09.005056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.004793 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:09.115669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.109430 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:09.115669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.109497 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 22:29:09.115669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.109813 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:09.435674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.433075 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:29:09.435674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.433135 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:29:09.483669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.482908 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" event={"ID":"debadcac-80e7-4843-a84a-dc60f364566c","Type":"ContainerStarted","Data":"f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8"} Nov 03 22:29:09.483669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.482974 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" event={"ID":"debadcac-80e7-4843-a84a-dc60f364566c","Type":"ContainerStarted","Data":"3f14b1c4fa78cfa9c707c53f7bce18165f759d8e95c84b92de8d82b1388c289e"} Nov 03 22:29:09.499727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.499051 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:09.658750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:09.658304 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" podStartSLOduration=1.658267012 podStartE2EDuration="1.658267012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:09.623114961 +0000 UTC m=+1306.603848619" watchObservedRunningTime="2026-11-03 22:29:09.658267012 +0000 UTC m=+1306.639000665" Nov 03 22:29:10.099667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:10.094237 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:10.106669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:10.102078 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:10.106669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:10.102150 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 22:29:10.506017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:10.447771 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:29:10.506017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:10.447834 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:29:10.506017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:10.505015 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 22:29:10.606263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:10.606217 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:10.753756 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:10.752264 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:29:10.753756 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:10.752325 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 22:29:10.806877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:10.781398 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.026804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.026757 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.107403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.107301 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:11.110786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.110728 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 22:29:11.234993 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.234936 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:29:11.291853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.289964 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.336223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.336169 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:29:11.485049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.484997 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.564927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.553323 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:11.569687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.565541 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.602667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.569955 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:29:11.665610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.665570 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:29:11.665927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.665895 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:29:11.666088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.666072 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:11.725057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.674140 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 22:29:11.725057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.682998 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.725057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.684328 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:29:11.725057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.684389 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 22:29:11.736723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.733063 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.806676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.805917 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:11.843667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.843501 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.859669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.844439 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.993729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:11.966894 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:12.086975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.040354 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.143948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.070412 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-kube-api-access-hdbm6\" (UniqueName: \"kubernetes.io/projected/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-kube-api-access-hdbm6\") pod \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " Nov 03 22:29:12.144292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.144270 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-openshift-ca\") pod \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " Nov 03 22:29:12.211223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.211179 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-kube-api-access-hdbm6" pod "0234c2e8-ecd4-4b8e-a523-4cab2501ad3c" (UID: "0234c2e8-ecd4-4b8e-a523-4cab2501ad3c"). InnerVolumeSpecName "kube-api-access-hdbm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:12.226028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.211957 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:12.226289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.226256 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 22:29:12.293244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.227473 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-xks-ca\") pod \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\" (UID: \"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c\") " Nov 03 22:29:12.293626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.293606 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044db9d5-efe9-4592-b923-4c86c8203a2a-kube-api-access-s5zqb\" (UniqueName: \"kubernetes.io/projected/044db9d5-efe9-4592-b923-4c86c8203a2a-kube-api-access-s5zqb\") pod \"044db9d5-efe9-4592-b923-4c86c8203a2a\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " Nov 03 22:29:12.300098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.300071 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-openshift-ca\") pod \"044db9d5-efe9-4592-b923-4c86c8203a2a\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " Nov 03 22:29:12.301940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.301827 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdbm6\" (UniqueName: \"kubernetes.io/projected/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-kube-api-access-hdbm6\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:12.302429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.291972 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-xks-ca" pod "0234c2e8-ecd4-4b8e-a523-4cab2501ad3c" (UID: "0234c2e8-ecd4-4b8e-a523-4cab2501ad3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:12.310743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.310699 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-openshift-ca" pod "0234c2e8-ecd4-4b8e-a523-4cab2501ad3c" (UID: "0234c2e8-ecd4-4b8e-a523-4cab2501ad3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:12.311867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.291806 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.322105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.322057 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-openshift-ca" pod "044db9d5-efe9-4592-b923-4c86c8203a2a" (UID: "044db9d5-efe9-4592-b923-4c86c8203a2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:12.442706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.428587 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-xks-ca\") pod \"044db9d5-efe9-4592-b923-4c86c8203a2a\" (UID: \"044db9d5-efe9-4592-b923-4c86c8203a2a\") " Nov 03 22:29:12.474495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.434432 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:29:12.488835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.488789 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:29:12.489747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.451403 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044db9d5-efe9-4592-b923-4c86c8203a2a-kube-api-access-s5zqb" pod "044db9d5-efe9-4592-b923-4c86c8203a2a" (UID: "044db9d5-efe9-4592-b923-4c86c8203a2a"). InnerVolumeSpecName "kube-api-access-s5zqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:12.496626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.496508 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:29:12.496626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.496568 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 22:29:12.569869 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.569779 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:12.570400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.570378 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5zqb\" (UniqueName: \"kubernetes.io/projected/044db9d5-efe9-4592-b923-4c86c8203a2a-kube-api-access-s5zqb\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:12.570521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.570508 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:12.570608 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.570597 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:12.605438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.604945 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.605752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.605705 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:29:12.607232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.605916 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:29:12.615158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.614774 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.663707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.658084 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-xks-ca" pod "044db9d5-efe9-4592-b923-4c86c8203a2a" (UID: "044db9d5-efe9-4592-b923-4c86c8203a2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:12.669667 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:29:12.665878 4298 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.444s" Nov 03 22:29:12.674451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.670085 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:29:12.674695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.671915 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044db9d5-efe9-4592-b923-4c86c8203a2a-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:12.674932 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.674915 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:29:12.675101 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.675079 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:29:12.694666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.681799 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:29:12.694666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.681875 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:29:12.694666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.681914 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:29:12.694666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.681973 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:29:12.694666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.682024 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:29:12.694666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.682060 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:29:12.694666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.688227 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.694666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.692730 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 22:29:12.698663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.696712 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded" Nov 03 22:29:12.779846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.779800 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.781477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.781438 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:29:12.900863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.874571 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.901142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:12.878896 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.086392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.086360 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:29:13.089801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.089775 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:29:13.159726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.128949 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:13.173936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.173878 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 22:29:13.174250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.154752 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:13.175974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.175937 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.176686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.176667 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:29:13.244710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.240792 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.284421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.280444 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" Nov 03 22:29:13.394766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.329745 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:29:13.421421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.421332 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:29:13.421705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.421673 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:29:13.492351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.492311 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:29:13.590883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.590843 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l" event={"ID":"044db9d5-efe9-4592-b923-4c86c8203a2a","Type":"ContainerDied","Data":"32bb8fef3f737ec7f750e832fca941ba1635deedf7f1053efbc94a72c0ff0f15"} Nov 03 22:29:13.591429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.591409 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32bb8fef3f737ec7f750e832fca941ba1635deedf7f1053efbc94a72c0ff0f15" Nov 03 22:29:13.591557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.591540 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" event={"ID":"0234c2e8-ecd4-4b8e-a523-4cab2501ad3c","Type":"ContainerDied","Data":"143b6b08399b259357290e20516a83a4fe766a53fb5227efde0eabfb3fadc85b"} Nov 03 22:29:13.591677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.591660 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="143b6b08399b259357290e20516a83a4fe766a53fb5227efde0eabfb3fadc85b" Nov 03 22:29:13.591762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.536668 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:29:13.591895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.591877 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 22:29:13.591998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.521826 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8" Nov 03 22:29:13.622984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.622937 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.623575 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.623545 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:29:13.648669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.623738 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.648669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.638223 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.710063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.706559 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.808469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.808425 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.868048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.868019 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:29:13.901776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.901732 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.902294 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:13.902270 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 22:29:14.006110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.006053 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.247665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.236755 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Nov 03 22:29:14.267740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.267427 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 22:29:14.330320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.330263 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:14.346797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.346752 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 22:29:14.347100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.332681 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:29:14.382299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.382238 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.416071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.416040 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:29:14.418503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.418477 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:29:14.442389 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:29:14.442307 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0234c2e8_ecd4_4b8e_a523_4cab2501ad3c.slice/crio-143b6b08399b259357290e20516a83a4fe766a53fb5227efde0eabfb3fadc85b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0234c2e8_ecd4_4b8e_a523_4cab2501ad3c.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:29:14.448866 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.442524 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:29:14.452745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.452705 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:29:14.478033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.477996 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:29:14.478306 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.478276 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 22:29:14.493398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.493366 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:29:14.494085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.494063 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:29:14.553590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.553501 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:29:14.596323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:14.596292 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:29:14.650668 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:29:14.602314 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0234c2e8_ecd4_4b8e_a523_4cab2501ad3c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0234c2e8_ecd4_4b8e_a523_4cab2501ad3c.slice/crio-143b6b08399b259357290e20516a83a4fe766a53fb5227efde0eabfb3fadc85b\": RecentStats: unable to find data in memory cache]" Nov 03 22:29:15.106150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:15.106099 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:15.106364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:15.106197 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 22:29:15.113687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:15.112117 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9"] Nov 03 22:29:15.212668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:15.124845 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12fe3022-k8rh9"] Nov 03 22:29:15.297797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:15.297763 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355d6d35-1bf8-4e19-8eeb-ed6d6c026b65" path="/var/lib/kubelet/pods/355d6d35-1bf8-4e19-8eeb-ed6d6c026b65/volumes" Nov 03 22:29:15.420979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:15.420883 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:29:15.420979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:15.420933 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:29:15.484261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:15.484223 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:29:15.484596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:15.484565 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 22:29:16.108547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:16.108512 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:16.109194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:16.109156 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 22:29:16.421448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:16.421358 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:29:16.421603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:16.421438 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:29:16.503673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:16.498953 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:29:16.503673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:16.499009 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 22:29:16.745669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:16.744668 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:17.102606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.102514 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:17.102606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.102581 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 22:29:17.165736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.165701 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc"] Nov 03 22:29:17.166589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.166568 4298 cpu_manager.go:414] "containerMap: removing container" podUID="044db9d5-efe9-4592-b923-4c86c8203a2a" containerName="revoke-keys" Nov 03 22:29:17.166747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.166698 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044db9d5-efe9-4592-b923-4c86c8203a2a" containerName="revoke-keys" Nov 03 22:29:17.166814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.166783 4298 cpu_manager.go:414] "containerMap: removing container" podUID="0234c2e8-ecd4-4b8e-a523-4cab2501ad3c" containerName="revoke-keys" Nov 03 22:29:17.166814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.166793 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0234c2e8-ecd4-4b8e-a523-4cab2501ad3c" containerName="revoke-keys" Nov 03 22:29:17.166976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.166961 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="0234c2e8-ecd4-4b8e-a523-4cab2501ad3c" containerName="revoke-keys" Nov 03 22:29:17.167036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.166990 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="044db9d5-efe9-4592-b923-4c86c8203a2a" containerName="revoke-keys" Nov 03 22:29:17.171433 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.171412 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:17.245670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.239906 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc"] Nov 03 22:29:17.246014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.240100 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:17.246150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.240862 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:17.265669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.264571 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faa7d467-4lfcc\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:17.265669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.264746 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mhcf\" (UniqueName: \"kubernetes.io/projected/04001e8f-c750-4377-be93-7f865cc614fd-kube-api-access-7mhcf\") pod \"maas-api-revoke-sub-faa7d467-4lfcc\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:17.265669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.264817 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faa7d467-4lfcc\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:17.367083 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.366988 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faa7d467-4lfcc\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:17.367423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.367396 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhcf\" (UniqueName: \"kubernetes.io/projected/04001e8f-c750-4377-be93-7f865cc614fd-kube-api-access-7mhcf\") pod \"maas-api-revoke-sub-faa7d467-4lfcc\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:17.368456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.368434 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faa7d467-4lfcc\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:17.369374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.369352 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faa7d467-4lfcc\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:17.374121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.374099 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faa7d467-4lfcc\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:17.420404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.420366 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:29:17.420597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.420431 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:29:17.460667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.459944 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mhcf\" (UniqueName: \"kubernetes.io/projected/04001e8f-c750-4377-be93-7f865cc614fd-kube-api-access-7mhcf\") pod \"maas-api-revoke-sub-faa7d467-4lfcc\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:17.477694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.477039 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:29:17.477694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.477097 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 22:29:17.522708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:17.522665 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:17.610667 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:29:17.610386 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:29:18.019123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.013482 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc"] Nov 03 22:29:18.452611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.452538 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:18.453226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.453121 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:18.454134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.454115 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:29:18.477592 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.477554 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:29:18.477797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.477672 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 22:29:18.497245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.497207 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:18.497711 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.497680 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:18.498755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.498733 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:29:18.726310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.726212 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" event={"ID":"04001e8f-c750-4377-be93-7f865cc614fd","Type":"ContainerStarted","Data":"0d456ceebfa2f0f20e87815d808c1aed54635258d6b13a551fb409bbc1b02772"} Nov 03 22:29:18.726310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.726275 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" event={"ID":"04001e8f-c750-4377-be93-7f865cc614fd","Type":"ContainerStarted","Data":"85e21201cf03ba5c115bec339eaa4bf139fa7c53ce3ea1216a59cf09342b51f7"} Nov 03 22:29:18.972877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:18.972822 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" podStartSLOduration=1.972807425 podStartE2EDuration="1.972807425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:18.969946116 +0000 UTC m=+1315.950679776" watchObservedRunningTime="2026-11-03 22:29:18.972807425 +0000 UTC m=+1315.953541078" Nov 03 22:29:19.717479 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:19.717444 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:19.718136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:19.718112 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:19.719347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:19.719330 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:29:19.740174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:19.740138 4298 generic.go:356] "Generic (PLEG): container finished" podID="04001e8f-c750-4377-be93-7f865cc614fd" containerID="0d456ceebfa2f0f20e87815d808c1aed54635258d6b13a551fb409bbc1b02772" exitCode=0 Nov 03 22:29:19.740367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:19.740330 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" event={"ID":"04001e8f-c750-4377-be93-7f865cc614fd","Type":"ContainerDied","Data":"0d456ceebfa2f0f20e87815d808c1aed54635258d6b13a551fb409bbc1b02772"} Nov 03 22:29:20.899870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:20.899841 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:20.949932 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:20.949893 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-xks-ca\") pod \"04001e8f-c750-4377-be93-7f865cc614fd\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " Nov 03 22:29:20.950083 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:20.949964 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04001e8f-c750-4377-be93-7f865cc614fd-kube-api-access-7mhcf\" (UniqueName: \"kubernetes.io/projected/04001e8f-c750-4377-be93-7f865cc614fd-kube-api-access-7mhcf\") pod \"04001e8f-c750-4377-be93-7f865cc614fd\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " Nov 03 22:29:20.950119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:20.950080 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-openshift-ca\") pod \"04001e8f-c750-4377-be93-7f865cc614fd\" (UID: \"04001e8f-c750-4377-be93-7f865cc614fd\") " Nov 03 22:29:20.950474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:20.950438 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-openshift-ca" pod "04001e8f-c750-4377-be93-7f865cc614fd" (UID: "04001e8f-c750-4377-be93-7f865cc614fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:20.950598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:20.950578 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:20.952074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:20.952049 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-xks-ca" pod "04001e8f-c750-4377-be93-7f865cc614fd" (UID: "04001e8f-c750-4377-be93-7f865cc614fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:20.952188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:20.952075 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04001e8f-c750-4377-be93-7f865cc614fd-kube-api-access-7mhcf" pod "04001e8f-c750-4377-be93-7f865cc614fd" (UID: "04001e8f-c750-4377-be93-7f865cc614fd"). InnerVolumeSpecName "kube-api-access-7mhcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:21.051668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.051575 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04001e8f-c750-4377-be93-7f865cc614fd-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:21.051668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.051605 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mhcf\" (UniqueName: \"kubernetes.io/projected/04001e8f-c750-4377-be93-7f865cc614fd-kube-api-access-7mhcf\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:21.399198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.399103 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:21.756399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.756363 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" event={"ID":"04001e8f-c750-4377-be93-7f865cc614fd","Type":"ContainerDied","Data":"85e21201cf03ba5c115bec339eaa4bf139fa7c53ce3ea1216a59cf09342b51f7"} Nov 03 22:29:21.756555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.756416 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85e21201cf03ba5c115bec339eaa4bf139fa7c53ce3ea1216a59cf09342b51f7" Nov 03 22:29:21.756555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.756377 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc" Nov 03 22:29:21.890625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.890592 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m"] Nov 03 22:29:21.891187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.891173 4298 cpu_manager.go:414] "containerMap: removing container" podUID="04001e8f-c750-4377-be93-7f865cc614fd" containerName="revoke-keys" Nov 03 22:29:21.891246 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.891189 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04001e8f-c750-4377-be93-7f865cc614fd" containerName="revoke-keys" Nov 03 22:29:21.891310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.891300 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="04001e8f-c750-4377-be93-7f865cc614fd" containerName="revoke-keys" Nov 03 22:29:21.894879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.894862 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:21.941155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.941123 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj"] Nov 03 22:29:21.945104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.945086 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:21.948449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.948430 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:21.950393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.950371 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m"] Nov 03 22:29:21.963810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.963786 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec23852e-mwv2m\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:21.963923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.963828 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec23852e-mwv2m\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:21.964040 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.964015 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj9ht\" (UniqueName: \"kubernetes.io/projected/2123055a-b7be-4c6d-b422-a44ad5460f42-kube-api-access-bj9ht\") pod \"maas-api-revoke-sub-ec23852e-mwv2m\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:21.987806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:21.987776 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:22.053721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.053621 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj"] Nov 03 22:29:22.065031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.064990 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec23852e-mwv2m\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:22.065193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.065072 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec23852e-mwv2m\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:22.065193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.065113 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.065193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.065156 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.065193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.065186 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.065386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.065257 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.065386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.065302 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.065386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.065333 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsz9q\" (UniqueName: \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-kube-api-access-bsz9q\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.065386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.065358 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac669eb9-9135-44df-858a-7fd3c7238e28-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.065593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.065418 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj9ht\" (UniqueName: \"kubernetes.io/projected/2123055a-b7be-4c6d-b422-a44ad5460f42-kube-api-access-bj9ht\") pod \"maas-api-revoke-sub-ec23852e-mwv2m\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:22.065593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.065491 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.065593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.065571 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.066556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.066521 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec23852e-mwv2m\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:22.068452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.068424 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec23852e-mwv2m\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:22.125157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.125123 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj9ht\" (UniqueName: \"kubernetes.io/projected/2123055a-b7be-4c6d-b422-a44ad5460f42-kube-api-access-bj9ht\") pod \"maas-api-revoke-sub-ec23852e-mwv2m\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:22.166497 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.166460 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.166713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.166539 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.166713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.166639 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.166713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.166703 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.166912 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.166735 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.166912 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.166801 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.166912 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.166844 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.166912 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.166877 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsz9q\" (UniqueName: \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-kube-api-access-bsz9q\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.166912 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.166904 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac669eb9-9135-44df-858a-7fd3c7238e28-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.167157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.166926 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.167316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.167294 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.167568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.167545 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.167617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.167565 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.167795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.167777 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac669eb9-9135-44df-858a-7fd3c7238e28-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.169105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.169083 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.169459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.169442 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.208139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.208108 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:22.225319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.225286 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsz9q\" (UniqueName: \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-kube-api-access-bsz9q\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.225514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.225490 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.258431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.258397 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:22.579108 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:29:22.579079 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2123055a_b7be_4c6d_b422_a44ad5460f42.slice/crio-43fac0ca73bf8133aee1b3402df5e7ff980a451868b55eef695f331d1acb2477 WatchSource:0}: Error finding container 43fac0ca73bf8133aee1b3402df5e7ff980a451868b55eef695f331d1acb2477: Status 404 returned error can't find the container with id 43fac0ca73bf8133aee1b3402df5e7ff980a451868b55eef695f331d1acb2477 Nov 03 22:29:22.582521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.582501 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m"] Nov 03 22:29:22.690731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.690702 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj"] Nov 03 22:29:22.726166 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:29:22.726129 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac669eb9_9135_44df_858a_7fd3c7238e28.slice/crio-8900ddfe153355fd388ac723b4ede9a0b9d6f402523f2ad98e84d6ab7809b38a WatchSource:0}: Error finding container 8900ddfe153355fd388ac723b4ede9a0b9d6f402523f2ad98e84d6ab7809b38a: Status 404 returned error can't find the container with id 8900ddfe153355fd388ac723b4ede9a0b9d6f402523f2ad98e84d6ab7809b38a Nov 03 22:29:22.728858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.728827 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:22.728935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.728896 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:22.728935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.728925 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:22.764844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.764813 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" event={"ID":"ac669eb9-9135-44df-858a-7fd3c7238e28","Type":"ContainerStarted","Data":"8900ddfe153355fd388ac723b4ede9a0b9d6f402523f2ad98e84d6ab7809b38a"} Nov 03 22:29:22.766261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:22.766234 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" event={"ID":"2123055a-b7be-4c6d-b422-a44ad5460f42","Type":"ContainerStarted","Data":"43fac0ca73bf8133aee1b3402df5e7ff980a451868b55eef695f331d1acb2477"} Nov 03 22:29:23.365406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:23.365372 4298 scope.go:122] "RemoveContainer" containerID="4470d055c2e43db974128524efd969ca7a97f6a216848131eec5f64592b7f1e8" Nov 03 22:29:23.375433 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:23.375409 4298 scope.go:122] "RemoveContainer" containerID="5db33207d28f5960862ecdf5f275f3e28c78feaef75f29cd6858f8b3e563abb2" Nov 03 22:29:23.384736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:23.384713 4298 scope.go:122] "RemoveContainer" containerID="a232adaf186bb49ba9bc3c8a7396bd12c5f62dba9747311a759ee4b7fc5082ea" Nov 03 22:29:23.394247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:23.394226 4298 scope.go:122] "RemoveContainer" containerID="a3426280a127f5974d1366e5812c1569743b9beeee81a12c036f06be33eabb28" Nov 03 22:29:23.403047 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:23.403025 4298 scope.go:122] "RemoveContainer" containerID="04c504c40e72613c137665178ebf4cb8345d6ba882614037f721840274a39406" Nov 03 22:29:23.411745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:23.411722 4298 scope.go:122] "RemoveContainer" containerID="e8a5274b4a9f825bc51cbe162877828d089c3a10812e1a467fc67c8e474ae503" Nov 03 22:29:23.784889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:23.784850 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" event={"ID":"ac669eb9-9135-44df-858a-7fd3c7238e28","Type":"ContainerStarted","Data":"634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd"} Nov 03 22:29:23.786449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:23.786422 4298 generic.go:356] "Generic (PLEG): container finished" podID="2123055a-b7be-4c6d-b422-a44ad5460f42" containerID="4c25d6e0a139f4fc251501243556c9e4b85a77a35a2b3290119aa52b71d397dc" exitCode=0 Nov 03 22:29:23.786579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:23.786501 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" event={"ID":"2123055a-b7be-4c6d-b422-a44ad5460f42","Type":"ContainerDied","Data":"4c25d6e0a139f4fc251501243556c9e4b85a77a35a2b3290119aa52b71d397dc"} Nov 03 22:29:24.018295 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:24.018239 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" podStartSLOduration=3.018224705 podStartE2EDuration="3.018224705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:23.966078905 +0000 UTC m=+1320.946812558" watchObservedRunningTime="2026-11-03 22:29:24.018224705 +0000 UTC m=+1320.998958358" Nov 03 22:29:24.258832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:24.258791 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:24.260943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:24.260920 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:24.261029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:24.260977 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:29:25.015112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.015085 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:25.106577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.106479 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2123055a-b7be-4c6d-b422-a44ad5460f42-kube-api-access-bj9ht\" (UniqueName: \"kubernetes.io/projected/2123055a-b7be-4c6d-b422-a44ad5460f42-kube-api-access-bj9ht\") pod \"2123055a-b7be-4c6d-b422-a44ad5460f42\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " Nov 03 22:29:25.106862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.106625 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-openshift-ca\") pod \"2123055a-b7be-4c6d-b422-a44ad5460f42\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " Nov 03 22:29:25.106862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.106767 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-xks-ca\") pod \"2123055a-b7be-4c6d-b422-a44ad5460f42\" (UID: \"2123055a-b7be-4c6d-b422-a44ad5460f42\") " Nov 03 22:29:25.106996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.106941 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-openshift-ca" pod "2123055a-b7be-4c6d-b422-a44ad5460f42" (UID: "2123055a-b7be-4c6d-b422-a44ad5460f42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:25.107281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.107259 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:25.108906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.108866 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-xks-ca" pod "2123055a-b7be-4c6d-b422-a44ad5460f42" (UID: "2123055a-b7be-4c6d-b422-a44ad5460f42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:25.109095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.109069 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2123055a-b7be-4c6d-b422-a44ad5460f42-kube-api-access-bj9ht" pod "2123055a-b7be-4c6d-b422-a44ad5460f42" (UID: "2123055a-b7be-4c6d-b422-a44ad5460f42"). InnerVolumeSpecName "kube-api-access-bj9ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:25.208298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.208261 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj9ht\" (UniqueName: \"kubernetes.io/projected/2123055a-b7be-4c6d-b422-a44ad5460f42-kube-api-access-bj9ht\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:25.208298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.208293 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2123055a-b7be-4c6d-b422-a44ad5460f42-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:25.259137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.259102 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:25.259303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.259159 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:29:25.800979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.800940 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" Nov 03 22:29:25.801158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.800977 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m" event={"ID":"2123055a-b7be-4c6d-b422-a44ad5460f42","Type":"ContainerDied","Data":"43fac0ca73bf8133aee1b3402df5e7ff980a451868b55eef695f331d1acb2477"} Nov 03 22:29:25.801158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:25.801025 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43fac0ca73bf8133aee1b3402df5e7ff980a451868b55eef695f331d1acb2477" Nov 03 22:29:26.019261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:26.016095 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:26.259365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:26.259325 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:26.259515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:26.259381 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:29:27.259136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:27.259085 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:27.259538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:27.259152 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:29:28.260391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.260355 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:28.260919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.260417 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:29:28.275564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.275538 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q"] Nov 03 22:29:28.276130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.276104 4298 cpu_manager.go:414] "containerMap: removing container" podUID="2123055a-b7be-4c6d-b422-a44ad5460f42" containerName="revoke-keys" Nov 03 22:29:28.276130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.276133 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2123055a-b7be-4c6d-b422-a44ad5460f42" containerName="revoke-keys" Nov 03 22:29:28.276257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.276248 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="2123055a-b7be-4c6d-b422-a44ad5460f42" containerName="revoke-keys" Nov 03 22:29:28.279824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.279808 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:28.343335 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.343305 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:28.348072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.348043 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q"] Nov 03 22:29:28.369226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.369199 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqfr\" (UniqueName: \"kubernetes.io/projected/288a6772-58f1-4966-ac60-20f47af51fc1-kube-api-access-xbqfr\") pod \"maas-api-revoke-sub-9d23cb25-fd94q\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:28.369490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.369475 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d23cb25-fd94q\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:28.369760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.369746 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d23cb25-fd94q\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:28.393166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.393133 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:28.471006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.470975 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d23cb25-fd94q\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:28.471344 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.471316 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqfr\" (UniqueName: \"kubernetes.io/projected/288a6772-58f1-4966-ac60-20f47af51fc1-kube-api-access-xbqfr\") pod \"maas-api-revoke-sub-9d23cb25-fd94q\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:28.471513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.471494 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d23cb25-fd94q\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:28.471821 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.471792 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d23cb25-fd94q\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:28.474375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.474353 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d23cb25-fd94q\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:28.552232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.552086 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqfr\" (UniqueName: \"kubernetes.io/projected/288a6772-58f1-4966-ac60-20f47af51fc1-kube-api-access-xbqfr\") pod \"maas-api-revoke-sub-9d23cb25-fd94q\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:28.593257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:28.593222 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:28.997790 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:29:28.997760 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288a6772_58f1_4966_ac60_20f47af51fc1.slice/crio-e09fe8ee9721369217644e1dfc1ab0c0d13b076fa2ed6198eb86986bd5a94fa5 WatchSource:0}: Error finding container e09fe8ee9721369217644e1dfc1ab0c0d13b076fa2ed6198eb86986bd5a94fa5: Status 404 returned error can't find the container with id e09fe8ee9721369217644e1dfc1ab0c0d13b076fa2ed6198eb86986bd5a94fa5 Nov 03 22:29:29.003627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:29.003601 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q"] Nov 03 22:29:29.259782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:29.259697 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:29.259782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:29.259749 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:29:29.833907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:29.833856 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" event={"ID":"288a6772-58f1-4966-ac60-20f47af51fc1","Type":"ContainerStarted","Data":"168d7c448cf65b3d9066915189c4ffbbb801b2bbd1d37bfce9fd0be06403900c"} Nov 03 22:29:29.834395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:29.833927 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" event={"ID":"288a6772-58f1-4966-ac60-20f47af51fc1","Type":"ContainerStarted","Data":"e09fe8ee9721369217644e1dfc1ab0c0d13b076fa2ed6198eb86986bd5a94fa5"} Nov 03 22:29:29.955953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:29.955901 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" podStartSLOduration=1.955881658 podStartE2EDuration="1.955881658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:29.954924297 +0000 UTC m=+1326.935657949" watchObservedRunningTime="2026-11-03 22:29:29.955881658 +0000 UTC m=+1326.936615311" Nov 03 22:29:30.263864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:30.263834 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:30.264275 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:30.264248 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:30.265028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:30.265008 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:29:30.682342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:30.682257 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:30.842033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:30.841998 4298 generic.go:356] "Generic (PLEG): container finished" podID="288a6772-58f1-4966-ac60-20f47af51fc1" containerID="168d7c448cf65b3d9066915189c4ffbbb801b2bbd1d37bfce9fd0be06403900c" exitCode=0 Nov 03 22:29:30.842395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:30.842084 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" event={"ID":"288a6772-58f1-4966-ac60-20f47af51fc1","Type":"ContainerDied","Data":"168d7c448cf65b3d9066915189c4ffbbb801b2bbd1d37bfce9fd0be06403900c"} Nov 03 22:29:32.004215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.004190 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:32.121244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.121155 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-openshift-ca\") pod \"288a6772-58f1-4966-ac60-20f47af51fc1\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " Nov 03 22:29:32.121244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.121235 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288a6772-58f1-4966-ac60-20f47af51fc1-kube-api-access-xbqfr\" (UniqueName: \"kubernetes.io/projected/288a6772-58f1-4966-ac60-20f47af51fc1-kube-api-access-xbqfr\") pod \"288a6772-58f1-4966-ac60-20f47af51fc1\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " Nov 03 22:29:32.121477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.121288 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-xks-ca\") pod \"288a6772-58f1-4966-ac60-20f47af51fc1\" (UID: \"288a6772-58f1-4966-ac60-20f47af51fc1\") " Nov 03 22:29:32.121588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.121561 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-openshift-ca" pod "288a6772-58f1-4966-ac60-20f47af51fc1" (UID: "288a6772-58f1-4966-ac60-20f47af51fc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:32.121930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.121897 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:32.123620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.123581 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-xks-ca" pod "288a6772-58f1-4966-ac60-20f47af51fc1" (UID: "288a6772-58f1-4966-ac60-20f47af51fc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:32.123843 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.123822 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288a6772-58f1-4966-ac60-20f47af51fc1-kube-api-access-xbqfr" pod "288a6772-58f1-4966-ac60-20f47af51fc1" (UID: "288a6772-58f1-4966-ac60-20f47af51fc1"). InnerVolumeSpecName "kube-api-access-xbqfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:32.223049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.223012 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbqfr\" (UniqueName: \"kubernetes.io/projected/288a6772-58f1-4966-ac60-20f47af51fc1-kube-api-access-xbqfr\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:32.223049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.223044 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288a6772-58f1-4966-ac60-20f47af51fc1-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:32.857017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.856983 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" Nov 03 22:29:32.857201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.857016 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q" event={"ID":"288a6772-58f1-4966-ac60-20f47af51fc1","Type":"ContainerDied","Data":"e09fe8ee9721369217644e1dfc1ab0c0d13b076fa2ed6198eb86986bd5a94fa5"} Nov 03 22:29:32.857201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:32.857068 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e09fe8ee9721369217644e1dfc1ab0c0d13b076fa2ed6198eb86986bd5a94fa5" Nov 03 22:29:35.609017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:35.607758 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:35.630668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:35.628823 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 22:29:35.630668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:35.629265 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:29:35.630668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:35.630379 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:35.703706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:35.702772 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:35.713690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:35.713249 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 22:29:35.779858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:35.779811 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 22:29:35.817668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:35.816840 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:35.998847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:35.998746 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:36.048669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.047772 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.176404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.172278 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7"] Nov 03 22:29:36.176404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.173148 4298 cpu_manager.go:414] "containerMap: removing container" podUID="288a6772-58f1-4966-ac60-20f47af51fc1" containerName="revoke-keys" Nov 03 22:29:36.176404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.173172 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288a6772-58f1-4966-ac60-20f47af51fc1" containerName="revoke-keys" Nov 03 22:29:36.176404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.173342 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="288a6772-58f1-4966-ac60-20f47af51fc1" containerName="revoke-keys" Nov 03 22:29:36.253034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.252926 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.256006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.255975 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.271936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.271897 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7"] Nov 03 22:29:36.389071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.389033 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.421668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.396764 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.422442 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.422401 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.430666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.422797 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.455455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.430937 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.455990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.455957 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdnw\" (UniqueName: \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-kube-api-access-6pdnw\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.482920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.482874 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.497684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.483328 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.498637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.498605 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c643fd-8534-4e35-808f-6d40aaa2e04c-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.629911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.613481 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.655851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.655801 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.656194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.656170 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.687847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.687800 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.688693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.688665 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdnw\" (UniqueName: \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-kube-api-access-6pdnw\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.688898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.688879 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.689068 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.689051 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.689178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.689165 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c643fd-8534-4e35-808f-6d40aaa2e04c-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.704669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.689322 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.705062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.688467 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.705211 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.640370 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:29:36.709668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.706110 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:29:36.817871 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.817841 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924"] Nov 03 22:29:36.846713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.707387 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.856103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.640219 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:29:36.856103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.640181 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.856103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.708315 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c643fd-8534-4e35-808f-6d40aaa2e04c-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.856103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.660085 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.856103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.640340 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 22:29:36.856103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.708377 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.856103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.727906 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.856103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.793558 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.856103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.817747 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.856103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.707060 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.875979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.875939 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.920725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.916154 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:36.928948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.928914 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:29:36.938669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.929319 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:29:36.939008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.938986 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:29:36.940034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.940017 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:29:37.012336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.997838 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:29:37.012336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.005746 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:29:37.012336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.005840 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:29:37.012336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.990416 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdnw\" (UniqueName: \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-kube-api-access-6pdnw\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:37.012336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.929242 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.033718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:36.991174 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:37.033718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.016620 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924"] Nov 03 22:29:37.046456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.046414 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.124250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.119362 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.124250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.119407 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjcnd\" (UniqueName: \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-kube-api-access-pjcnd\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.124250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.119441 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.124250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.119467 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.124250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.119537 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.124250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.119690 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.124250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.119729 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea941933-edc9-4938-b3e6-42caa269c62b-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.124250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.119798 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea941933-edc9-4938-b3e6-42caa269c62b-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.225204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.224917 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:37.263775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.233180 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.263775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.233307 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.263775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.233340 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjcnd\" (UniqueName: \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-kube-api-access-pjcnd\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.263775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.233372 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.263775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.233400 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.263775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.233439 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.263775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.233530 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.263775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.233566 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea941933-edc9-4938-b3e6-42caa269c62b-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.263775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.233602 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea941933-edc9-4938-b3e6-42caa269c62b-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.263775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.234245 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 22:29:37.263775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.256003 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:37.329725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.272871 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.377837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.377791 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea941933-edc9-4938-b3e6-42caa269c62b-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.419428 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.419395 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.459789 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.427248 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 22:29:37.481688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.461864 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:29:37.544072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.529162 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.666943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.559955 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:37.749803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.749772 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:29:37.776689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.731791 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:37.908793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.778099 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:37.909087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.886847 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:37.934670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.887597 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:29:37.935269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.887625 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 22:29:37.935707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.887661 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 22:29:37.936049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.887680 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 22:29:37.936397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:37.887697 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 22:29:38.028728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.010263 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea941933-edc9-4938-b3e6-42caa269c62b-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:38.034963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.031864 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:38.088978 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.059237 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:38.089236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.059268 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5"] Nov 03 22:29:38.097978 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.097946 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:38.176034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.175993 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5"] Nov 03 22:29:38.177048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.177022 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjcnd\" (UniqueName: \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-kube-api-access-pjcnd\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:38.210208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.204536 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:38.210208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.205623 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:38.264370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.260981 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:38.264370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.263431 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:29:38.264637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.264392 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:38.298465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.266175 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1308704b-rjdt5\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:38.298465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.266215 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dxtk\" (UniqueName: \"kubernetes.io/projected/bd349022-5e19-4264-85d7-c8bb5244a0b2-kube-api-access-6dxtk\") pod \"maas-api-revoke-sub-1308704b-rjdt5\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:38.298465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.266346 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1308704b-rjdt5\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:38.349700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.347992 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:38.360217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.359520 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:29:38.379991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.366765 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:38.379991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.367668 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1308704b-rjdt5\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:38.379991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.367716 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dxtk\" (UniqueName: \"kubernetes.io/projected/bd349022-5e19-4264-85d7-c8bb5244a0b2-kube-api-access-6dxtk\") pod \"maas-api-revoke-sub-1308704b-rjdt5\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:38.379991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.367919 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1308704b-rjdt5\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:38.411968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.411923 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1308704b-rjdt5\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:38.412901 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.412879 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1308704b-rjdt5\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:38.435159 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.435116 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:38.437559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.437529 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:29:38.644698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.617484 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dxtk\" (UniqueName: \"kubernetes.io/projected/bd349022-5e19-4264-85d7-c8bb5244a0b2-kube-api-access-6dxtk\") pod \"maas-api-revoke-sub-1308704b-rjdt5\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:38.661679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:38.655671 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:39.008455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.008376 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:39.027172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.027119 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:29:39.030024 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.029989 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:29:39.031283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.031242 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 22:29:39.032295 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.032271 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 22:29:39.153259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.130831 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:29:39.317853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.266925 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:39.392532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.351895 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 22:29:39.392532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.392501 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:39.450508 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.448968 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx"] Nov 03 22:29:39.458470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.458437 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:29:39.458676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.458520 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:29:39.458676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.458545 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx"] Nov 03 22:29:39.458808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.458677 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:29:39.458808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.458725 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:29:39.458808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.458805 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:29:39.458971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.458839 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:29:39.458971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.458891 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:29:39.460372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.460340 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.472099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.470894 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:29:39.498117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.498087 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:39.498267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.498252 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:39.498357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.498347 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:39.498862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.498840 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7"] Nov 03 22:29:39.588415 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.588381 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.588601 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.588439 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.588707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.588596 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.588707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.588673 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e55d3001-b0d1-41a6-a376-8f057710801b-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.588890 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.588868 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.589013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.588990 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.589816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.589257 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.589935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.589888 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9j76\" (UniqueName: \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-kube-api-access-g9j76\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.589935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.589928 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e55d3001-b0d1-41a6-a376-8f057710801b-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.691210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.691171 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.691368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.691342 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9j76\" (UniqueName: \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-kube-api-access-g9j76\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.691409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.691375 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e55d3001-b0d1-41a6-a376-8f057710801b-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.691481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.691464 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.691481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.691467 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.691588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.691539 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.691662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.691602 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.691716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.691632 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e55d3001-b0d1-41a6-a376-8f057710801b-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.691716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.691703 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.691832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.691765 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.691962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.691939 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.692144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.692122 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.692298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.692255 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.692589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.692541 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e55d3001-b0d1-41a6-a376-8f057710801b-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.694193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.694168 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e55d3001-b0d1-41a6-a376-8f057710801b-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.694368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.694344 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.749324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.749288 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.749503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.749480 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9j76\" (UniqueName: \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-kube-api-access-g9j76\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.788463 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.788415 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:39.796881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.796753 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5"] Nov 03 22:29:39.844095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.843511 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:39.844095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.843597 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:39.844095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.843658 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:39.850323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:39.850293 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924"] Nov 03 22:29:40.195504 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:29:40.195455 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55d3001_b0d1_41a6_a376_8f057710801b.slice/crio-760cc8603bf2f9c8f77cd770f70a39b0b2c60cfb43299128f966d6e6c2833746 WatchSource:0}: Error finding container 760cc8603bf2f9c8f77cd770f70a39b0b2c60cfb43299128f966d6e6c2833746: Status 404 returned error can't find the container with id 760cc8603bf2f9c8f77cd770f70a39b0b2c60cfb43299128f966d6e6c2833746 Nov 03 22:29:40.198877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.198844 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:40.198977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.198913 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:40.198977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.198949 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:40.200581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.200555 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx"] Nov 03 22:29:40.483624 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.483577 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" event={"ID":"ea941933-edc9-4938-b3e6-42caa269c62b","Type":"ContainerStarted","Data":"cad9cfe1c5a0da8bbea8d0a4c4521af0313527b387c2f34941702c5bec5e4ec9"} Nov 03 22:29:40.483996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.483973 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" event={"ID":"ea941933-edc9-4938-b3e6-42caa269c62b","Type":"ContainerStarted","Data":"de582f4f37a9ec778003e87a74369a002e8b8756a65df80e98a85a7b854dd6e4"} Nov 03 22:29:40.486920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.486891 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" event={"ID":"e55d3001-b0d1-41a6-a376-8f057710801b","Type":"ContainerStarted","Data":"3945917e0bb4418c951fa3a3dc4f9b80ac4801067e203b8f73086f42e146faf9"} Nov 03 22:29:40.487054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.486945 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" event={"ID":"e55d3001-b0d1-41a6-a376-8f057710801b","Type":"ContainerStarted","Data":"760cc8603bf2f9c8f77cd770f70a39b0b2c60cfb43299128f966d6e6c2833746"} Nov 03 22:29:40.489290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.489263 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" event={"ID":"12c643fd-8534-4e35-808f-6d40aaa2e04c","Type":"ContainerStarted","Data":"b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969"} Nov 03 22:29:40.489424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.489311 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" event={"ID":"12c643fd-8534-4e35-808f-6d40aaa2e04c","Type":"ContainerStarted","Data":"5227f7a40b85289fe5b1e11f21980f5ff61868e81e9b537d3c88157b6d84dbfc"} Nov 03 22:29:40.491034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.491011 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" event={"ID":"bd349022-5e19-4264-85d7-c8bb5244a0b2","Type":"ContainerStarted","Data":"0a144e5f2a6d56f481e2f6a520ea9d2ff6853479b97980fe54b0d1f7e96ff182"} Nov 03 22:29:40.491126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.491056 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" event={"ID":"bd349022-5e19-4264-85d7-c8bb5244a0b2","Type":"ContainerStarted","Data":"881f929755be8ba52c2aa06a586bfb374d467e28c4cd6348e4d46ce3ade6e8b1"} Nov 03 22:29:40.789828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:40.789740 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:41.311792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.302798 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:29:41.311792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.302868 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 22:29:41.311792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.303187 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:29:41.311792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.303221 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" podUID="12c643fd-8534-4e35-808f-6d40aaa2e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 22:29:41.316723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.316694 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:41.316881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.316742 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:41.508209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.488952 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" podStartSLOduration=3.488932298 podStartE2EDuration="3.488932298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:41.348116172 +0000 UTC m=+1338.328849826" watchObservedRunningTime="2026-11-03 22:29:41.488932298 +0000 UTC m=+1338.469665952" Nov 03 22:29:41.519952 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.519923 4298 generic.go:356] "Generic (PLEG): container finished" podID="bd349022-5e19-4264-85d7-c8bb5244a0b2" containerID="0a144e5f2a6d56f481e2f6a520ea9d2ff6853479b97980fe54b0d1f7e96ff182" exitCode=0 Nov 03 22:29:41.520152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.520130 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" event={"ID":"bd349022-5e19-4264-85d7-c8bb5244a0b2","Type":"ContainerDied","Data":"0a144e5f2a6d56f481e2f6a520ea9d2ff6853479b97980fe54b0d1f7e96ff182"} Nov 03 22:29:41.612911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.612805 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" podStartSLOduration=5.612788887 podStartE2EDuration="5.612788887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:41.488046264 +0000 UTC m=+1338.468779918" watchObservedRunningTime="2026-11-03 22:29:41.612788887 +0000 UTC m=+1338.593522539" Nov 03 22:29:41.613236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.613209 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" podStartSLOduration=4.613203689 podStartE2EDuration="4.613203689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:41.612291585 +0000 UTC m=+1338.593025238" watchObservedRunningTime="2026-11-03 22:29:41.613203689 +0000 UTC m=+1338.593937339" Nov 03 22:29:41.797669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.797542 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:41.797669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.797604 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:29:41.811662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.798014 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc"] Nov 03 22:29:41.811662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.804217 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" podStartSLOduration=5.804197369 podStartE2EDuration="5.804197369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:41.761239305 +0000 UTC m=+1338.741972983" watchObservedRunningTime="2026-11-03 22:29:41.804197369 +0000 UTC m=+1338.784931025" Nov 03 22:29:41.842670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.842003 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:41.921774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:41.921685 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc"] Nov 03 22:29:42.013944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.009519 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc67462e-vklrc\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:42.013944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.009620 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc67462e-vklrc\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:42.013944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.009838 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9xf\" (UniqueName: \"kubernetes.io/projected/e5aad511-b51a-4e53-9259-1865e2d67b61-kube-api-access-qs9xf\") pod \"maas-api-revoke-sub-bc67462e-vklrc\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:42.112712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.112532 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9xf\" (UniqueName: \"kubernetes.io/projected/e5aad511-b51a-4e53-9259-1865e2d67b61-kube-api-access-qs9xf\") pod \"maas-api-revoke-sub-bc67462e-vklrc\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:42.113096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.113045 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc67462e-vklrc\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:42.113596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.113567 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc67462e-vklrc\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:42.113780 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.113756 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc67462e-vklrc\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:42.116124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.116102 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc67462e-vklrc\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:42.204120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.204038 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9xf\" (UniqueName: \"kubernetes.io/projected/e5aad511-b51a-4e53-9259-1865e2d67b61-kube-api-access-qs9xf\") pod \"maas-api-revoke-sub-bc67462e-vklrc\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:42.244085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.243975 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:42.249689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.248248 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:29:42.249689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.248307 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 22:29:42.249689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.248790 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:29:42.249689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.248828 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" podUID="12c643fd-8534-4e35-808f-6d40aaa2e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 22:29:42.310615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.310581 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:42.814338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.814252 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:42.814894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:42.814864 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:29:43.275235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.275193 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:29:43.275485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.275455 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 22:29:43.276824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.276800 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:29:43.276985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.276963 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" podUID="12c643fd-8534-4e35-808f-6d40aaa2e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 22:29:43.370893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.337052 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:43.409651 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.409616 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc"] Nov 03 22:29:43.421435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.421405 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-openshift-ca\") pod \"bd349022-5e19-4264-85d7-c8bb5244a0b2\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " Nov 03 22:29:43.421529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.421455 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd349022-5e19-4264-85d7-c8bb5244a0b2-kube-api-access-6dxtk\" (UniqueName: \"kubernetes.io/projected/bd349022-5e19-4264-85d7-c8bb5244a0b2-kube-api-access-6dxtk\") pod \"bd349022-5e19-4264-85d7-c8bb5244a0b2\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " Nov 03 22:29:43.421602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.421589 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-xks-ca\") pod \"bd349022-5e19-4264-85d7-c8bb5244a0b2\" (UID: \"bd349022-5e19-4264-85d7-c8bb5244a0b2\") " Nov 03 22:29:43.421950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.421917 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-openshift-ca" pod "bd349022-5e19-4264-85d7-c8bb5244a0b2" (UID: "bd349022-5e19-4264-85d7-c8bb5244a0b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:43.422078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.422063 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:43.479665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.463560 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd349022-5e19-4264-85d7-c8bb5244a0b2-kube-api-access-6dxtk" pod "bd349022-5e19-4264-85d7-c8bb5244a0b2" (UID: "bd349022-5e19-4264-85d7-c8bb5244a0b2"). InnerVolumeSpecName "kube-api-access-6dxtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:43.515667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.513154 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-xks-ca" pod "bd349022-5e19-4264-85d7-c8bb5244a0b2" (UID: "bd349022-5e19-4264-85d7-c8bb5244a0b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:43.535669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.523766 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd349022-5e19-4264-85d7-c8bb5244a0b2-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:43.535669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.523803 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dxtk\" (UniqueName: \"kubernetes.io/projected/bd349022-5e19-4264-85d7-c8bb5244a0b2-kube-api-access-6dxtk\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:43.615583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.615352 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" event={"ID":"e5aad511-b51a-4e53-9259-1865e2d67b61","Type":"ContainerStarted","Data":"771f411f4f0677f54916cd050640564ec91b1d9d2a854a7db9a89e29364cf6f5"} Nov 03 22:29:43.616683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.616659 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" Nov 03 22:29:43.616683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.616672 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5" event={"ID":"bd349022-5e19-4264-85d7-c8bb5244a0b2","Type":"ContainerDied","Data":"881f929755be8ba52c2aa06a586bfb374d467e28c4cd6348e4d46ce3ade6e8b1"} Nov 03 22:29:43.616847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.616698 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="881f929755be8ba52c2aa06a586bfb374d467e28c4cd6348e4d46ce3ade6e8b1" Nov 03 22:29:43.798669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.791862 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:43.798669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:43.791927 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:29:44.264714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:44.247107 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:29:44.264714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:44.247177 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 22:29:44.264714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:44.247899 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:29:44.264714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:44.247983 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" podUID="12c643fd-8534-4e35-808f-6d40aaa2e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 22:29:44.660260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:44.660225 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" event={"ID":"e5aad511-b51a-4e53-9259-1865e2d67b61","Type":"ContainerStarted","Data":"9122ae334a2886aebc35c6954b87a7de3f190d0be5347a25dc0c2e76a9116469"} Nov 03 22:29:44.800114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:44.800073 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:44.800435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:44.800404 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:29:44.806956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:44.806900 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" podStartSLOduration=3.806886638 podStartE2EDuration="3.806886638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:44.786145937 +0000 UTC m=+1341.766879616" watchObservedRunningTime="2026-11-03 22:29:44.806886638 +0000 UTC m=+1341.787620290" Nov 03 22:29:45.244191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:45.244154 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:29:45.244522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:45.244477 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 22:29:45.248175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:45.248076 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:29:45.248175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:45.248135 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" podUID="12c643fd-8534-4e35-808f-6d40aaa2e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 22:29:45.681689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:45.681100 4298 generic.go:356] "Generic (PLEG): container finished" podID="e5aad511-b51a-4e53-9259-1865e2d67b61" containerID="9122ae334a2886aebc35c6954b87a7de3f190d0be5347a25dc0c2e76a9116469" exitCode=0 Nov 03 22:29:45.681689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:45.681178 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" event={"ID":"e5aad511-b51a-4e53-9259-1865e2d67b61","Type":"ContainerDied","Data":"9122ae334a2886aebc35c6954b87a7de3f190d0be5347a25dc0c2e76a9116469"} Nov 03 22:29:45.806936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:45.806897 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:45.807144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:45.806980 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:29:46.252677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.252305 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:29:46.252677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.252380 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 22:29:46.252956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.252830 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:29:46.252956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.252870 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" podUID="12c643fd-8534-4e35-808f-6d40aaa2e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 22:29:46.793943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.791267 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:46.793943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.791327 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:29:46.894570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.894540 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:46.914767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.914733 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-xks-ca\") pod \"e5aad511-b51a-4e53-9259-1865e2d67b61\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " Nov 03 22:29:46.914988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.914812 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-openshift-ca\") pod \"e5aad511-b51a-4e53-9259-1865e2d67b61\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " Nov 03 22:29:46.914988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.914847 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5aad511-b51a-4e53-9259-1865e2d67b61-kube-api-access-qs9xf\" (UniqueName: \"kubernetes.io/projected/e5aad511-b51a-4e53-9259-1865e2d67b61-kube-api-access-qs9xf\") pod \"e5aad511-b51a-4e53-9259-1865e2d67b61\" (UID: \"e5aad511-b51a-4e53-9259-1865e2d67b61\") " Nov 03 22:29:46.915537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.915503 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-openshift-ca" pod "e5aad511-b51a-4e53-9259-1865e2d67b61" (UID: "e5aad511-b51a-4e53-9259-1865e2d67b61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:46.918038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.918005 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-xks-ca" pod "e5aad511-b51a-4e53-9259-1865e2d67b61" (UID: "e5aad511-b51a-4e53-9259-1865e2d67b61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:46.918184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:46.918030 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5aad511-b51a-4e53-9259-1865e2d67b61-kube-api-access-qs9xf" pod "e5aad511-b51a-4e53-9259-1865e2d67b61" (UID: "e5aad511-b51a-4e53-9259-1865e2d67b61"). InnerVolumeSpecName "kube-api-access-qs9xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:47.020769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.017981 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:47.020769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.018023 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aad511-b51a-4e53-9259-1865e2d67b61-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:47.020769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.018041 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs9xf\" (UniqueName: \"kubernetes.io/projected/e5aad511-b51a-4e53-9259-1865e2d67b61-kube-api-access-qs9xf\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:47.245851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.245707 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 22:29:47.245851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.245786 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 22:29:47.568215 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:29:47.568173 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:29:47.681670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.681278 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:47.683061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.682859 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:47.686020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.685990 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:29:47.698401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.698371 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" Nov 03 22:29:47.698565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.698378 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc" event={"ID":"e5aad511-b51a-4e53-9259-1865e2d67b61","Type":"ContainerDied","Data":"771f411f4f0677f54916cd050640564ec91b1d9d2a854a7db9a89e29364cf6f5"} Nov 03 22:29:47.698565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.698533 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="771f411f4f0677f54916cd050640564ec91b1d9d2a854a7db9a89e29364cf6f5" Nov 03 22:29:47.789313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.789276 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:47.789458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:47.789344 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:29:48.249217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:48.249187 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:48.249697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:48.249678 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:48.250551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:48.250527 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:29:48.794213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:48.794181 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:48.794679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:48.794619 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:48.795405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:48.795381 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:29:49.264569 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:49.264513 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:49.454911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:49.454870 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:51.270043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.269996 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g"] Nov 03 22:29:51.270570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.270555 4298 cpu_manager.go:414] "containerMap: removing container" podUID="bd349022-5e19-4264-85d7-c8bb5244a0b2" containerName="revoke-keys" Nov 03 22:29:51.270616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.270570 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd349022-5e19-4264-85d7-c8bb5244a0b2" containerName="revoke-keys" Nov 03 22:29:51.270616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.270581 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e5aad511-b51a-4e53-9259-1865e2d67b61" containerName="revoke-keys" Nov 03 22:29:51.270616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.270586 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5aad511-b51a-4e53-9259-1865e2d67b61" containerName="revoke-keys" Nov 03 22:29:51.270768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.270747 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd349022-5e19-4264-85d7-c8bb5244a0b2" containerName="revoke-keys" Nov 03 22:29:51.270800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.270769 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5aad511-b51a-4e53-9259-1865e2d67b61" containerName="revoke-keys" Nov 03 22:29:51.274421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.274404 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.391928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.391893 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.391928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.391937 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.392166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.391956 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.392166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.392135 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.392265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.392189 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189b8cc5-c678-4a68-81b9-144c292c18af-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.392322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.392306 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.392406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.392383 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.392547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.392503 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcl9\" (UniqueName: \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-kube-api-access-6wcl9\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.392654 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.392559 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189b8cc5-c678-4a68-81b9-144c292c18af-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.481627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.481596 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g"] Nov 03 22:29:51.494242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.494210 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.494411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.494284 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189b8cc5-c678-4a68-81b9-144c292c18af-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.494411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.494341 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.494411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.494360 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.494411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.494391 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcl9\" (UniqueName: \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-kube-api-access-6wcl9\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.494576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.494410 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189b8cc5-c678-4a68-81b9-144c292c18af-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.494576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.494443 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.494576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.494462 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.494576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.494478 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.494834 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.494708 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.494913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.494891 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.495582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.495555 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189b8cc5-c678-4a68-81b9-144c292c18af-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.495802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.495775 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.496027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.496008 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.497227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.497205 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.497512 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.497481 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189b8cc5-c678-4a68-81b9-144c292c18af-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.554605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.554535 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.554787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.554765 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcl9\" (UniqueName: \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-kube-api-access-6wcl9\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.564854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.564823 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn"] Nov 03 22:29:51.570387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.570315 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:51.587422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.587393 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:51.624513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.624255 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:51.625064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.625044 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:51.627669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.627602 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn"] Nov 03 22:29:51.696982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.696946 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dbd6b9-khjnn\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:51.697201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.697180 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndqp\" (UniqueName: \"kubernetes.io/projected/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-kube-api-access-tndqp\") pod \"maas-api-revoke-sub-83dbd6b9-khjnn\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:51.697321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.697298 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dbd6b9-khjnn\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:51.798336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.798288 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dbd6b9-khjnn\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:51.798538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.798414 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dbd6b9-khjnn\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:51.798616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.798562 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndqp\" (UniqueName: \"kubernetes.io/projected/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-kube-api-access-tndqp\") pod \"maas-api-revoke-sub-83dbd6b9-khjnn\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:51.799008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.798979 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dbd6b9-khjnn\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:51.800787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.800769 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dbd6b9-khjnn\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:51.857542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.857471 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndqp\" (UniqueName: \"kubernetes.io/projected/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-kube-api-access-tndqp\") pod \"maas-api-revoke-sub-83dbd6b9-khjnn\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:51.870087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.870055 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:51.895983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:51.895955 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:52.019782 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:29:52.019738 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189b8cc5_c678_4a68_81b9_144c292c18af.slice/crio-cd15fb10f45d370eb96242a7940128f81e6995bf7ad222831aa41c53ba8214e9 WatchSource:0}: Error finding container cd15fb10f45d370eb96242a7940128f81e6995bf7ad222831aa41c53ba8214e9: Status 404 returned error can't find the container with id cd15fb10f45d370eb96242a7940128f81e6995bf7ad222831aa41c53ba8214e9 Nov 03 22:29:52.023199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.023164 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:52.023318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.023252 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:52.023318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.023298 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:29:52.024745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.024721 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g"] Nov 03 22:29:52.284398 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:29:52.284362 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef819e5c_60a5_48f2_b5e9_ea8ca06aa8a9.slice/crio-8a0453539335ede8b1b3005e4b366432179259b03e88677869e483611412e495 WatchSource:0}: Error finding container 8a0453539335ede8b1b3005e4b366432179259b03e88677869e483611412e495: Status 404 returned error can't find the container with id 8a0453539335ede8b1b3005e4b366432179259b03e88677869e483611412e495 Nov 03 22:29:52.288499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.288475 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn"] Nov 03 22:29:52.514761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.514725 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd"] Nov 03 22:29:52.523371 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.523336 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:52.590700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.590639 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd"] Nov 03 22:29:52.608369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.608321 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfxr\" (UniqueName: \"kubernetes.io/projected/157b49b4-5df7-421e-9b9b-5318bdba773e-kube-api-access-lnfxr\") pod \"maas-api-revoke-sub-f806c996-996nd\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:52.608538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.608409 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f806c996-996nd\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:52.608757 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.608729 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f806c996-996nd\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:52.710239 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.710203 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f806c996-996nd\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:52.710446 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.710285 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfxr\" (UniqueName: \"kubernetes.io/projected/157b49b4-5df7-421e-9b9b-5318bdba773e-kube-api-access-lnfxr\") pod \"maas-api-revoke-sub-f806c996-996nd\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:52.710446 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.710341 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f806c996-996nd\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:52.710971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.710942 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f806c996-996nd\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:52.712721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.712703 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f806c996-996nd\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:52.756767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.756733 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" event={"ID":"189b8cc5-c678-4a68-81b9-144c292c18af","Type":"ContainerStarted","Data":"c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec"} Nov 03 22:29:52.756897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.756788 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" event={"ID":"189b8cc5-c678-4a68-81b9-144c292c18af","Type":"ContainerStarted","Data":"cd15fb10f45d370eb96242a7940128f81e6995bf7ad222831aa41c53ba8214e9"} Nov 03 22:29:52.758517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.758492 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" event={"ID":"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9","Type":"ContainerStarted","Data":"8a0453539335ede8b1b3005e4b366432179259b03e88677869e483611412e495"} Nov 03 22:29:52.768883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.768859 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfxr\" (UniqueName: \"kubernetes.io/projected/157b49b4-5df7-421e-9b9b-5318bdba773e-kube-api-access-lnfxr\") pod \"maas-api-revoke-sub-f806c996-996nd\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:52.864105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.864021 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:52.877248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:52.877202 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" podStartSLOduration=1.877187186 podStartE2EDuration="1.877187186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:52.876965155 +0000 UTC m=+1349.857698809" watchObservedRunningTime="2026-11-03 22:29:52.877187186 +0000 UTC m=+1349.857920839" Nov 03 22:29:53.252841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:53.252808 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd"] Nov 03 22:29:53.253721 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:29:53.253691 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157b49b4_5df7_421e_9b9b_5318bdba773e.slice/crio-a6ebc653790709450c1cb33960f63051182d901d4c4e6d9c1956927b87196fda WatchSource:0}: Error finding container a6ebc653790709450c1cb33960f63051182d901d4c4e6d9c1956927b87196fda: Status 404 returned error can't find the container with id a6ebc653790709450c1cb33960f63051182d901d4c4e6d9c1956927b87196fda Nov 03 22:29:53.587998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:53.587959 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:53.589480 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:53.589455 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:53.589554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:53.589509 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:29:53.766928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:53.766836 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" event={"ID":"157b49b4-5df7-421e-9b9b-5318bdba773e","Type":"ContainerStarted","Data":"9cd7f1f6548eca7b6d97f6d48eafc0c31aafc04a9115e4c728a515295ab01641"} Nov 03 22:29:53.766928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:53.766898 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" event={"ID":"157b49b4-5df7-421e-9b9b-5318bdba773e","Type":"ContainerStarted","Data":"a6ebc653790709450c1cb33960f63051182d901d4c4e6d9c1956927b87196fda"} Nov 03 22:29:53.768391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:53.768363 4298 generic.go:356] "Generic (PLEG): container finished" podID="ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9" containerID="bccf2eb16385e75d427f433f2f566e2814056e7886bea0e4a228f9265e68a105" exitCode=0 Nov 03 22:29:53.768511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:53.768446 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" event={"ID":"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9","Type":"ContainerDied","Data":"bccf2eb16385e75d427f433f2f566e2814056e7886bea0e4a228f9265e68a105"} Nov 03 22:29:53.887736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:53.887682 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" podStartSLOduration=1.887668355 podStartE2EDuration="1.887668355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:53.885179058 +0000 UTC m=+1350.865912710" watchObservedRunningTime="2026-11-03 22:29:53.887668355 +0000 UTC m=+1350.868402002" Nov 03 22:29:54.587993 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:54.587959 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:54.588142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:54.588031 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:29:54.777829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:54.777738 4298 generic.go:356] "Generic (PLEG): container finished" podID="157b49b4-5df7-421e-9b9b-5318bdba773e" containerID="9cd7f1f6548eca7b6d97f6d48eafc0c31aafc04a9115e4c728a515295ab01641" exitCode=0 Nov 03 22:29:54.777829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:54.777820 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" event={"ID":"157b49b4-5df7-421e-9b9b-5318bdba773e","Type":"ContainerDied","Data":"9cd7f1f6548eca7b6d97f6d48eafc0c31aafc04a9115e4c728a515295ab01641"} Nov 03 22:29:54.977297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:54.977267 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:55.067759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.067678 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-openshift-ca\") pod \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " Nov 03 22:29:55.067909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.067793 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-kube-api-access-tndqp\" (UniqueName: \"kubernetes.io/projected/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-kube-api-access-tndqp\") pod \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " Nov 03 22:29:55.067952 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.067910 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-xks-ca\") pod \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\" (UID: \"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9\") " Nov 03 22:29:55.068115 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.068094 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-openshift-ca" pod "ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9" (UID: "ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:55.068433 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.068411 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:55.069853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.069830 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-xks-ca" pod "ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9" (UID: "ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:55.069950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.069913 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-kube-api-access-tndqp" pod "ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9" (UID: "ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9"). InnerVolumeSpecName "kube-api-access-tndqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:55.170568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.170535 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:55.170568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.170568 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tndqp\" (UniqueName: \"kubernetes.io/projected/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9-kube-api-access-tndqp\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:55.588547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.588508 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:55.588958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.588572 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:29:55.785977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.785937 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" Nov 03 22:29:55.786145 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.785969 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn" event={"ID":"ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9","Type":"ContainerDied","Data":"8a0453539335ede8b1b3005e4b366432179259b03e88677869e483611412e495"} Nov 03 22:29:55.786145 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.786034 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a0453539335ede8b1b3005e4b366432179259b03e88677869e483611412e495" Nov 03 22:29:55.990982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:55.990953 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:56.082898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.082860 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157b49b4-5df7-421e-9b9b-5318bdba773e-kube-api-access-lnfxr\" (UniqueName: \"kubernetes.io/projected/157b49b4-5df7-421e-9b9b-5318bdba773e-kube-api-access-lnfxr\") pod \"157b49b4-5df7-421e-9b9b-5318bdba773e\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " Nov 03 22:29:56.083149 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.083131 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-xks-ca\") pod \"157b49b4-5df7-421e-9b9b-5318bdba773e\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " Nov 03 22:29:56.083248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.083235 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-openshift-ca\") pod \"157b49b4-5df7-421e-9b9b-5318bdba773e\" (UID: \"157b49b4-5df7-421e-9b9b-5318bdba773e\") " Nov 03 22:29:56.083561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.083539 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-openshift-ca" pod "157b49b4-5df7-421e-9b9b-5318bdba773e" (UID: "157b49b4-5df7-421e-9b9b-5318bdba773e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:56.083705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.083687 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:56.085013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.084992 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-xks-ca" pod "157b49b4-5df7-421e-9b9b-5318bdba773e" (UID: "157b49b4-5df7-421e-9b9b-5318bdba773e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:56.085082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.085013 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157b49b4-5df7-421e-9b9b-5318bdba773e-kube-api-access-lnfxr" pod "157b49b4-5df7-421e-9b9b-5318bdba773e" (UID: "157b49b4-5df7-421e-9b9b-5318bdba773e"). InnerVolumeSpecName "kube-api-access-lnfxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:56.185101 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.185019 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157b49b4-5df7-421e-9b9b-5318bdba773e-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:56.185101 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.185049 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnfxr\" (UniqueName: \"kubernetes.io/projected/157b49b4-5df7-421e-9b9b-5318bdba773e-kube-api-access-lnfxr\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:56.547530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.547462 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:56.587998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.587966 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:56.588167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.588019 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:29:56.795500 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.795469 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" Nov 03 22:29:56.795940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.795507 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd" event={"ID":"157b49b4-5df7-421e-9b9b-5318bdba773e","Type":"ContainerDied","Data":"a6ebc653790709450c1cb33960f63051182d901d4c4e6d9c1956927b87196fda"} Nov 03 22:29:56.795940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:56.795564 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6ebc653790709450c1cb33960f63051182d901d4c4e6d9c1956927b87196fda" Nov 03 22:29:57.305230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.305192 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w"] Nov 03 22:29:57.305910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.305889 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9" containerName="revoke-keys" Nov 03 22:29:57.305910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.305911 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9" containerName="revoke-keys" Nov 03 22:29:57.306079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.305927 4298 cpu_manager.go:414] "containerMap: removing container" podUID="157b49b4-5df7-421e-9b9b-5318bdba773e" containerName="revoke-keys" Nov 03 22:29:57.306079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.305935 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157b49b4-5df7-421e-9b9b-5318bdba773e" containerName="revoke-keys" Nov 03 22:29:57.306173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.306150 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9" containerName="revoke-keys" Nov 03 22:29:57.306173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.306164 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="157b49b4-5df7-421e-9b9b-5318bdba773e" containerName="revoke-keys" Nov 03 22:29:57.310012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.309989 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:29:57.362124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.362092 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w"] Nov 03 22:29:57.370681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.365989 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:57.402196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.402162 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a73cd797-bf76w\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:29:57.402365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.402208 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a73cd797-bf76w\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:29:57.402365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.402318 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntng\" (UniqueName: \"kubernetes.io/projected/0ba540bf-4c1f-455c-a819-6b3e80c0d987-kube-api-access-qntng\") pod \"maas-api-revoke-sub-a73cd797-bf76w\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:29:57.408757 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.408734 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:57.503405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.503370 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a73cd797-bf76w\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:29:57.503405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.503412 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a73cd797-bf76w\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:29:57.503622 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.503455 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntng\" (UniqueName: \"kubernetes.io/projected/0ba540bf-4c1f-455c-a819-6b3e80c0d987-kube-api-access-qntng\") pod \"maas-api-revoke-sub-a73cd797-bf76w\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:29:57.504206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.504182 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a73cd797-bf76w\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:29:57.505848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.505826 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a73cd797-bf76w\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:29:57.562282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.562207 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntng\" (UniqueName: \"kubernetes.io/projected/0ba540bf-4c1f-455c-a819-6b3e80c0d987-kube-api-access-qntng\") pod \"maas-api-revoke-sub-a73cd797-bf76w\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:29:57.588551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.588511 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:57.588721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.588572 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:29:57.648696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:57.632878 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:29:58.022105 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:29:58.022062 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba540bf_4c1f_455c_a819_6b3e80c0d987.slice/crio-37f4ec7c30eea2529e0111ea188af8976f29c0e27c4cc260cb79dd646c1f4ecc WatchSource:0}: Error finding container 37f4ec7c30eea2529e0111ea188af8976f29c0e27c4cc260cb79dd646c1f4ecc: Status 404 returned error can't find the container with id 37f4ec7c30eea2529e0111ea188af8976f29c0e27c4cc260cb79dd646c1f4ecc Nov 03 22:29:58.025887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:58.025858 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w"] Nov 03 22:29:58.588098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:58.588006 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:58.588098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:58.588069 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:29:58.813892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:58.813851 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" event={"ID":"0ba540bf-4c1f-455c-a819-6b3e80c0d987","Type":"ContainerStarted","Data":"ba388dae3773e1bf2c328df1fc48b8734d0d88fa583d9f6232207595526e3564"} Nov 03 22:29:58.814097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:58.813910 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" event={"ID":"0ba540bf-4c1f-455c-a819-6b3e80c0d987","Type":"ContainerStarted","Data":"37f4ec7c30eea2529e0111ea188af8976f29c0e27c4cc260cb79dd646c1f4ecc"} Nov 03 22:29:58.945776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:58.945713 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" podStartSLOduration=1.945691392 podStartE2EDuration="1.945691392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:58.939283354 +0000 UTC m=+1355.920017022" watchObservedRunningTime="2026-11-03 22:29:58.945691392 +0000 UTC m=+1355.926425046" Nov 03 22:29:59.593000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:59.592969 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:59.593472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:59.593421 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:59.594291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:59.594269 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:29:59.824403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:59.824363 4298 generic.go:356] "Generic (PLEG): container finished" podID="0ba540bf-4c1f-455c-a819-6b3e80c0d987" containerID="ba388dae3773e1bf2c328df1fc48b8734d0d88fa583d9f6232207595526e3564" exitCode=0 Nov 03 22:29:59.824581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:29:59.824452 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" event={"ID":"0ba540bf-4c1f-455c-a819-6b3e80c0d987","Type":"ContainerDied","Data":"ba388dae3773e1bf2c328df1fc48b8734d0d88fa583d9f6232207595526e3564"} Nov 03 22:30:00.222294 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.222264 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r"] Nov 03 22:30:00.226556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.226532 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7"] Nov 03 22:30:00.226708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.226669 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" Nov 03 22:30:00.230665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.230631 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" Nov 03 22:30:00.246585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.246555 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk"] Nov 03 22:30:00.250822 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.250801 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" Nov 03 22:30:00.256955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.256932 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx"] Nov 03 22:30:00.261055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.261032 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" Nov 03 22:30:00.290145 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.290111 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7"] Nov 03 22:30:00.299009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.298967 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r"] Nov 03 22:30:00.305890 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.305847 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk"] Nov 03 22:30:00.318346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.318304 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx"] Nov 03 22:30:00.350736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.350700 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca5763c-a422-4d74-8571-1cf7f31a51ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7\" (UID: \"8ca5763c-a422-4d74-8571-1cf7f31a51ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" Nov 03 22:30:00.350922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.350760 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzmmb\" (UniqueName: \"kubernetes.io/projected/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-kube-api-access-zzmmb\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx\" (UID: \"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" Nov 03 22:30:00.350922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.350898 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z27jx\" (UniqueName: \"kubernetes.io/projected/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-kube-api-access-z27jx\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r\" (UID: \"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" Nov 03 22:30:00.351056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.351012 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r\" (UID: \"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" Nov 03 22:30:00.351113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.351064 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e758b-9785-4a8e-889c-70003899615f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk\" (UID: \"369e758b-9785-4a8e-889c-70003899615f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" Nov 03 22:30:00.351167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.351136 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx\" (UID: \"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" Nov 03 22:30:00.351455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.351412 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gspl8\" (UniqueName: \"kubernetes.io/projected/8ca5763c-a422-4d74-8571-1cf7f31a51ff-kube-api-access-gspl8\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7\" (UID: \"8ca5763c-a422-4d74-8571-1cf7f31a51ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" Nov 03 22:30:00.351580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.351521 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxkk\" (UniqueName: \"kubernetes.io/projected/369e758b-9785-4a8e-889c-70003899615f-kube-api-access-smxkk\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk\" (UID: \"369e758b-9785-4a8e-889c-70003899615f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" Nov 03 22:30:00.452792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.452757 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzmmb\" (UniqueName: \"kubernetes.io/projected/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-kube-api-access-zzmmb\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx\" (UID: \"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" Nov 03 22:30:00.452942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.452813 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27jx\" (UniqueName: \"kubernetes.io/projected/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-kube-api-access-z27jx\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r\" (UID: \"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" Nov 03 22:30:00.452942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.452861 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r\" (UID: \"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" Nov 03 22:30:00.452942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.452887 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e758b-9785-4a8e-889c-70003899615f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk\" (UID: \"369e758b-9785-4a8e-889c-70003899615f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" Nov 03 22:30:00.452942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.452937 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx\" (UID: \"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" Nov 03 22:30:00.453108 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.453077 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gspl8\" (UniqueName: \"kubernetes.io/projected/8ca5763c-a422-4d74-8571-1cf7f31a51ff-kube-api-access-gspl8\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7\" (UID: \"8ca5763c-a422-4d74-8571-1cf7f31a51ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" Nov 03 22:30:00.453166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.453149 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxkk\" (UniqueName: \"kubernetes.io/projected/369e758b-9785-4a8e-889c-70003899615f-kube-api-access-smxkk\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk\" (UID: \"369e758b-9785-4a8e-889c-70003899615f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" Nov 03 22:30:00.453240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.453213 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca5763c-a422-4d74-8571-1cf7f31a51ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7\" (UID: \"8ca5763c-a422-4d74-8571-1cf7f31a51ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" Nov 03 22:30:00.453616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.453582 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r\" (UID: \"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" Nov 03 22:30:00.453795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.453770 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx\" (UID: \"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" Nov 03 22:30:00.453850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.453832 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e758b-9785-4a8e-889c-70003899615f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk\" (UID: \"369e758b-9785-4a8e-889c-70003899615f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" Nov 03 22:30:00.453983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.453966 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca5763c-a422-4d74-8571-1cf7f31a51ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7\" (UID: \"8ca5763c-a422-4d74-8571-1cf7f31a51ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" Nov 03 22:30:00.511928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.511803 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gspl8\" (UniqueName: \"kubernetes.io/projected/8ca5763c-a422-4d74-8571-1cf7f31a51ff-kube-api-access-gspl8\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7\" (UID: \"8ca5763c-a422-4d74-8571-1cf7f31a51ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" Nov 03 22:30:00.511928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.511852 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27jx\" (UniqueName: \"kubernetes.io/projected/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-kube-api-access-z27jx\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r\" (UID: \"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" Nov 03 22:30:00.512190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.512163 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzmmb\" (UniqueName: \"kubernetes.io/projected/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-kube-api-access-zzmmb\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx\" (UID: \"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" Nov 03 22:30:00.512261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.512203 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxkk\" (UniqueName: \"kubernetes.io/projected/369e758b-9785-4a8e-889c-70003899615f-kube-api-access-smxkk\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk\" (UID: \"369e758b-9785-4a8e-889c-70003899615f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" Nov 03 22:30:00.542128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.542097 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" Nov 03 22:30:00.549585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.549556 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" Nov 03 22:30:00.564718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.564695 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" Nov 03 22:30:00.578951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.578923 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" Nov 03 22:30:00.941552 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:00.941516 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca5763c_a422_4d74_8571_1cf7f31a51ff.slice/crio-c959ba1a3c070f2b0b92a799e11b83cbd3934e81f497d14adf246fa5df363c5e WatchSource:0}: Error finding container c959ba1a3c070f2b0b92a799e11b83cbd3934e81f497d14adf246fa5df363c5e: Status 404 returned error can't find the container with id c959ba1a3c070f2b0b92a799e11b83cbd3934e81f497d14adf246fa5df363c5e Nov 03 22:30:00.954611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.954575 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7"] Nov 03 22:30:00.958687 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:00.958661 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b0ef3_a35f_4ea8_b7f2_c70e81a768b3.slice/crio-f337792f394d3b77b7f71f9ecc0e43adf5d25c3d54a7b9b7f467f59329c06f8d WatchSource:0}: Error finding container f337792f394d3b77b7f71f9ecc0e43adf5d25c3d54a7b9b7f467f59329c06f8d: Status 404 returned error can't find the container with id f337792f394d3b77b7f71f9ecc0e43adf5d25c3d54a7b9b7f467f59329c06f8d Nov 03 22:30:00.961372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.961352 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r"] Nov 03 22:30:00.982427 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:00.982392 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda207322_70a5_4bf3_a1ba_40ed3cfaf1d4.slice/crio-9a0cb38b5199883dd666763bb57e20a6aab85710021d013b015939ca572feeaf WatchSource:0}: Error finding container 9a0cb38b5199883dd666763bb57e20a6aab85710021d013b015939ca572feeaf: Status 404 returned error can't find the container with id 9a0cb38b5199883dd666763bb57e20a6aab85710021d013b015939ca572feeaf Nov 03 22:30:00.985889 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:00.985855 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369e758b_9785_4a8e_889c_70003899615f.slice/crio-e93e93c42a521ee700a0de9bf621ff9c83a7f5c1e67b2fe1b0cda49540a3e127 WatchSource:0}: Error finding container e93e93c42a521ee700a0de9bf621ff9c83a7f5c1e67b2fe1b0cda49540a3e127: Status 404 returned error can't find the container with id e93e93c42a521ee700a0de9bf621ff9c83a7f5c1e67b2fe1b0cda49540a3e127 Nov 03 22:30:00.988756 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.988684 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx"] Nov 03 22:30:00.990674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.990612 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:30:00.992940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:00.992912 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk"] Nov 03 22:30:01.140074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.134193 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:01.176255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.176221 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba540bf-4c1f-455c-a819-6b3e80c0d987-kube-api-access-qntng\" (UniqueName: \"kubernetes.io/projected/0ba540bf-4c1f-455c-a819-6b3e80c0d987-kube-api-access-qntng\") pod \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " Nov 03 22:30:01.176445 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.176346 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-openshift-ca\") pod \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " Nov 03 22:30:01.176555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.176539 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-xks-ca\") pod \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\" (UID: \"0ba540bf-4c1f-455c-a819-6b3e80c0d987\") " Nov 03 22:30:01.176829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.176801 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-openshift-ca" pod "0ba540bf-4c1f-455c-a819-6b3e80c0d987" (UID: "0ba540bf-4c1f-455c-a819-6b3e80c0d987"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:01.177123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.177101 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:01.228427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.228394 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-xks-ca" pod "0ba540bf-4c1f-455c-a819-6b3e80c0d987" (UID: "0ba540bf-4c1f-455c-a819-6b3e80c0d987"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:01.228609 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.228584 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba540bf-4c1f-455c-a819-6b3e80c0d987-kube-api-access-qntng" pod "0ba540bf-4c1f-455c-a819-6b3e80c0d987" (UID: "0ba540bf-4c1f-455c-a819-6b3e80c0d987"). InnerVolumeSpecName "kube-api-access-qntng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:01.278213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.278178 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba540bf-4c1f-455c-a819-6b3e80c0d987-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:01.278213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.278209 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qntng\" (UniqueName: \"kubernetes.io/projected/0ba540bf-4c1f-455c-a819-6b3e80c0d987-kube-api-access-qntng\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:01.842009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.841971 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" event={"ID":"8ca5763c-a422-4d74-8571-1cf7f31a51ff","Type":"ContainerStarted","Data":"f10098315a4584de50d6ba910b2a32bdfe803a442adcd7b4c35bb2e20525ce37"} Nov 03 22:30:01.842205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.842036 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" event={"ID":"8ca5763c-a422-4d74-8571-1cf7f31a51ff","Type":"ContainerStarted","Data":"c959ba1a3c070f2b0b92a799e11b83cbd3934e81f497d14adf246fa5df363c5e"} Nov 03 22:30:01.843699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.843672 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" event={"ID":"369e758b-9785-4a8e-889c-70003899615f","Type":"ContainerStarted","Data":"fb634e00f55b0cff7bb82034008b44241e0fc56f7befe6608e27382ab4a1fb00"} Nov 03 22:30:01.843796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.843721 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" event={"ID":"369e758b-9785-4a8e-889c-70003899615f","Type":"ContainerStarted","Data":"e93e93c42a521ee700a0de9bf621ff9c83a7f5c1e67b2fe1b0cda49540a3e127"} Nov 03 22:30:01.845258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.845236 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" event={"ID":"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4","Type":"ContainerStarted","Data":"4aaf71fa17310aa8f5b5cf6bebd9be460938eeea92b0029cbcd9d82f173cdabb"} Nov 03 22:30:01.845329 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.845281 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" event={"ID":"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4","Type":"ContainerStarted","Data":"9a0cb38b5199883dd666763bb57e20a6aab85710021d013b015939ca572feeaf"} Nov 03 22:30:01.846536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.846520 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" Nov 03 22:30:01.846536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.846524 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w" event={"ID":"0ba540bf-4c1f-455c-a819-6b3e80c0d987","Type":"ContainerDied","Data":"37f4ec7c30eea2529e0111ea188af8976f29c0e27c4cc260cb79dd646c1f4ecc"} Nov 03 22:30:01.846697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.846568 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f4ec7c30eea2529e0111ea188af8976f29c0e27c4cc260cb79dd646c1f4ecc" Nov 03 22:30:01.848146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.848123 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" event={"ID":"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3","Type":"ContainerStarted","Data":"07a03f9b72fed75b4343db1b61a778d6343a0a2cb783cc7b3c6903668ed86ddc"} Nov 03 22:30:01.848241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.848156 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" event={"ID":"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3","Type":"ContainerStarted","Data":"f337792f394d3b77b7f71f9ecc0e43adf5d25c3d54a7b9b7f467f59329c06f8d"} Nov 03 22:30:01.964397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:01.964334 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" podStartSLOduration=1.964318053 podStartE2EDuration="1.964318053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:01.958884412 +0000 UTC m=+1358.939618068" watchObservedRunningTime="2026-11-03 22:30:01.964318053 +0000 UTC m=+1358.945051713" Nov 03 22:30:02.076424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.076254 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" podStartSLOduration=2.076235197 podStartE2EDuration="2.076235197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:02.075150219 +0000 UTC m=+1359.055883872" watchObservedRunningTime="2026-11-03 22:30:02.076235197 +0000 UTC m=+1359.056968850" Nov 03 22:30:02.194677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.194600 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" podStartSLOduration=2.194579876 podStartE2EDuration="2.194579876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:02.191250429 +0000 UTC m=+1359.171984084" watchObservedRunningTime="2026-11-03 22:30:02.194579876 +0000 UTC m=+1359.175313531" Nov 03 22:30:02.311191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.311131 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" podStartSLOduration=2.3111156299999998 podStartE2EDuration="2.31111563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:02.308160955 +0000 UTC m=+1359.288894607" watchObservedRunningTime="2026-11-03 22:30:02.31111563 +0000 UTC m=+1359.291849283" Nov 03 22:30:02.858207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.858165 4298 generic.go:356] "Generic (PLEG): container finished" podID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerID="07a03f9b72fed75b4343db1b61a778d6343a0a2cb783cc7b3c6903668ed86ddc" exitCode=60 Nov 03 22:30:02.858367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.858256 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" event={"ID":"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3","Type":"ContainerDied","Data":"07a03f9b72fed75b4343db1b61a778d6343a0a2cb783cc7b3c6903668ed86ddc"} Nov 03 22:30:02.858660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.858623 4298 scope.go:122] "RemoveContainer" containerID="07a03f9b72fed75b4343db1b61a778d6343a0a2cb783cc7b3c6903668ed86ddc" Nov 03 22:30:02.860005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.859981 4298 generic.go:356] "Generic (PLEG): container finished" podID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerID="f10098315a4584de50d6ba910b2a32bdfe803a442adcd7b4c35bb2e20525ce37" exitCode=60 Nov 03 22:30:02.860089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.860057 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" event={"ID":"8ca5763c-a422-4d74-8571-1cf7f31a51ff","Type":"ContainerDied","Data":"f10098315a4584de50d6ba910b2a32bdfe803a442adcd7b4c35bb2e20525ce37"} Nov 03 22:30:02.860304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.860283 4298 scope.go:122] "RemoveContainer" containerID="f10098315a4584de50d6ba910b2a32bdfe803a442adcd7b4c35bb2e20525ce37" Nov 03 22:30:02.862131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.861935 4298 generic.go:356] "Generic (PLEG): container finished" podID="369e758b-9785-4a8e-889c-70003899615f" containerID="fb634e00f55b0cff7bb82034008b44241e0fc56f7befe6608e27382ab4a1fb00" exitCode=60 Nov 03 22:30:02.862131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.862016 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" event={"ID":"369e758b-9785-4a8e-889c-70003899615f","Type":"ContainerDied","Data":"fb634e00f55b0cff7bb82034008b44241e0fc56f7befe6608e27382ab4a1fb00"} Nov 03 22:30:02.862393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.862251 4298 scope.go:122] "RemoveContainer" containerID="fb634e00f55b0cff7bb82034008b44241e0fc56f7befe6608e27382ab4a1fb00" Nov 03 22:30:02.864217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.864198 4298 generic.go:356] "Generic (PLEG): container finished" podID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerID="4aaf71fa17310aa8f5b5cf6bebd9be460938eeea92b0029cbcd9d82f173cdabb" exitCode=60 Nov 03 22:30:02.864282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.864256 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" event={"ID":"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4","Type":"ContainerDied","Data":"4aaf71fa17310aa8f5b5cf6bebd9be460938eeea92b0029cbcd9d82f173cdabb"} Nov 03 22:30:02.864506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:02.864492 4298 scope.go:122] "RemoveContainer" containerID="4aaf71fa17310aa8f5b5cf6bebd9be460938eeea92b0029cbcd9d82f173cdabb" Nov 03 22:30:03.874992 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:03.874956 4298 generic.go:356] "Generic (PLEG): container finished" podID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerID="e77f3a4b56bade4037cd07cfe52aac83bb7ba4c5fa7a4d3c07e79fd7f423d0b1" exitCode=60 Nov 03 22:30:03.875449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:03.875031 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" event={"ID":"8ca5763c-a422-4d74-8571-1cf7f31a51ff","Type":"ContainerDied","Data":"e77f3a4b56bade4037cd07cfe52aac83bb7ba4c5fa7a4d3c07e79fd7f423d0b1"} Nov 03 22:30:03.875449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:03.875093 4298 scope.go:122] "RemoveContainer" containerID="f10098315a4584de50d6ba910b2a32bdfe803a442adcd7b4c35bb2e20525ce37" Nov 03 22:30:03.875449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:03.875406 4298 scope.go:122] "RemoveContainer" containerID="e77f3a4b56bade4037cd07cfe52aac83bb7ba4c5fa7a4d3c07e79fd7f423d0b1" Nov 03 22:30:03.875833 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:03.875804 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7_odh-ai-gateway-infra(8ca5763c-a422-4d74-8571-1cf7f31a51ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" Nov 03 22:30:03.877360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:03.877336 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" event={"ID":"369e758b-9785-4a8e-889c-70003899615f","Type":"ContainerStarted","Data":"a56f895e58fb4761d840024a7b130a62668a40c18fe5124de7258c3ee9910e78"} Nov 03 22:30:03.879598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:03.879568 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" event={"ID":"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4","Type":"ContainerStarted","Data":"96910ba1b1bcec70b57d823c860f5dff8665be2737a23143212c9778841b13ca"} Nov 03 22:30:03.881761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:03.881721 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" event={"ID":"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3","Type":"ContainerStarted","Data":"7ef64f42d8c40126fb6c0c9fcaf5f9ede86577523361e004a00374942e76ff6b"} Nov 03 22:30:03.985226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:03.985193 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:04.269381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.269349 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:04.788077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.788037 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:04.892508 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.892484 4298 scope.go:122] "RemoveContainer" containerID="e77f3a4b56bade4037cd07cfe52aac83bb7ba4c5fa7a4d3c07e79fd7f423d0b1" Nov 03 22:30:04.892971 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:04.892874 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7_odh-ai-gateway-infra(8ca5763c-a422-4d74-8571-1cf7f31a51ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" Nov 03 22:30:04.894167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.894141 4298 generic.go:356] "Generic (PLEG): container finished" podID="369e758b-9785-4a8e-889c-70003899615f" containerID="a56f895e58fb4761d840024a7b130a62668a40c18fe5124de7258c3ee9910e78" exitCode=60 Nov 03 22:30:04.894309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.894268 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" event={"ID":"369e758b-9785-4a8e-889c-70003899615f","Type":"ContainerDied","Data":"a56f895e58fb4761d840024a7b130a62668a40c18fe5124de7258c3ee9910e78"} Nov 03 22:30:04.894365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.894325 4298 scope.go:122] "RemoveContainer" containerID="fb634e00f55b0cff7bb82034008b44241e0fc56f7befe6608e27382ab4a1fb00" Nov 03 22:30:04.894554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.894523 4298 scope.go:122] "RemoveContainer" containerID="a56f895e58fb4761d840024a7b130a62668a40c18fe5124de7258c3ee9910e78" Nov 03 22:30:04.894928 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:04.894903 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk_odh-ai-gateway-infra(369e758b-9785-4a8e-889c-70003899615f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" podUID="369e758b-9785-4a8e-889c-70003899615f" Nov 03 22:30:04.898669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.897050 4298 generic.go:356] "Generic (PLEG): container finished" podID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerID="96910ba1b1bcec70b57d823c860f5dff8665be2737a23143212c9778841b13ca" exitCode=60 Nov 03 22:30:04.898669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.897213 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" event={"ID":"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4","Type":"ContainerDied","Data":"96910ba1b1bcec70b57d823c860f5dff8665be2737a23143212c9778841b13ca"} Nov 03 22:30:04.898669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.897718 4298 scope.go:122] "RemoveContainer" containerID="96910ba1b1bcec70b57d823c860f5dff8665be2737a23143212c9778841b13ca" Nov 03 22:30:04.898669 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:04.898152 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx_odh-ai-gateway-infra(da207322-70a5-4bf3-a1ba-40ed3cfaf1d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" Nov 03 22:30:04.902269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.902241 4298 generic.go:356] "Generic (PLEG): container finished" podID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerID="7ef64f42d8c40126fb6c0c9fcaf5f9ede86577523361e004a00374942e76ff6b" exitCode=60 Nov 03 22:30:04.902398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.902321 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" event={"ID":"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3","Type":"ContainerDied","Data":"7ef64f42d8c40126fb6c0c9fcaf5f9ede86577523361e004a00374942e76ff6b"} Nov 03 22:30:04.902625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.902610 4298 scope.go:122] "RemoveContainer" containerID="7ef64f42d8c40126fb6c0c9fcaf5f9ede86577523361e004a00374942e76ff6b" Nov 03 22:30:04.902987 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:04.902955 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r_odh-ai-gateway-infra(f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" Nov 03 22:30:04.912468 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.912445 4298 scope.go:122] "RemoveContainer" containerID="4aaf71fa17310aa8f5b5cf6bebd9be460938eeea92b0029cbcd9d82f173cdabb" Nov 03 22:30:04.926515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.926494 4298 scope.go:122] "RemoveContainer" containerID="07a03f9b72fed75b4343db1b61a778d6343a0a2cb783cc7b3c6903668ed86ddc" Nov 03 22:30:04.976573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:04.976541 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:05.282959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.282924 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:05.448024 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.447985 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m"] Nov 03 22:30:05.448717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.448695 4298 cpu_manager.go:414] "containerMap: removing container" podUID="0ba540bf-4c1f-455c-a819-6b3e80c0d987" containerName="revoke-keys" Nov 03 22:30:05.448717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.448716 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba540bf-4c1f-455c-a819-6b3e80c0d987" containerName="revoke-keys" Nov 03 22:30:05.448885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.448830 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba540bf-4c1f-455c-a819-6b3e80c0d987" containerName="revoke-keys" Nov 03 22:30:05.453722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.453693 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.625199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.625106 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m"] Nov 03 22:30:05.643025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.641316 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cee4c09d-openshift-default-dockercfg-7z5j6\"" Nov 03 22:30:05.644484 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.644445 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-data\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.644627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.644519 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-envoy\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.644784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.644710 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-credential-socket\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.644784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.644751 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-certs\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.644784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.644779 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh2cp\" (UniqueName: \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-kube-api-access-hh2cp\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.644953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.644892 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-podinfo\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.644953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.644924 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-token\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.645057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.645001 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-socket\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.645057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.645027 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istiod-ca-cert\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.653689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.652501 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:05.746888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.746805 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-credential-socket\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.746888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.746864 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-certs\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.746888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.746892 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh2cp\" (UniqueName: \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-kube-api-access-hh2cp\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.747229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.746986 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-podinfo\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.747229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.747188 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-token\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.747332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.747302 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-credential-socket\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.747332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.747313 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-socket\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.747438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.747360 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istiod-ca-cert\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.747503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.747461 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-data\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.747562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.747504 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-envoy\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.747825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.747785 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-data\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.748139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.748114 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-certs\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.748301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.748182 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istiod-ca-cert\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.748456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.748434 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-socket\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.751945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.751910 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-envoy\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.752778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.752744 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-podinfo\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.806458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.806415 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-token\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.806458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.806434 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh2cp\" (UniqueName: \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-kube-api-access-hh2cp\") pod \"e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:05.846481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.846436 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:05.918490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.918398 4298 scope.go:122] "RemoveContainer" containerID="a56f895e58fb4761d840024a7b130a62668a40c18fe5124de7258c3ee9910e78" Nov 03 22:30:05.919017 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:05.918884 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk_odh-ai-gateway-infra(369e758b-9785-4a8e-889c-70003899615f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" podUID="369e758b-9785-4a8e-889c-70003899615f" Nov 03 22:30:05.920357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.920327 4298 scope.go:122] "RemoveContainer" containerID="96910ba1b1bcec70b57d823c860f5dff8665be2737a23143212c9778841b13ca" Nov 03 22:30:05.920763 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:05.920717 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx_odh-ai-gateway-infra(da207322-70a5-4bf3-a1ba-40ed3cfaf1d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" Nov 03 22:30:05.922309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:05.922289 4298 scope.go:122] "RemoveContainer" containerID="7ef64f42d8c40126fb6c0c9fcaf5f9ede86577523361e004a00374942e76ff6b" Nov 03 22:30:05.922604 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:05.922583 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r_odh-ai-gateway-infra(f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" Nov 03 22:30:06.050092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.050048 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:06.068860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.068823 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:06.252729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.252116 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:06.443081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.443021 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:06.522328 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:06.522299 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f976bf_42fe_4c21_9408_30fc92f0d5bc.slice/crio-b5f3e3a4118d59983416627bcc91a162f11386090b33b15ad2c8ad639813bab5 WatchSource:0}: Error finding container b5f3e3a4118d59983416627bcc91a162f11386090b33b15ad2c8ad639813bab5: Status 404 returned error can't find the container with id b5f3e3a4118d59983416627bcc91a162f11386090b33b15ad2c8ad639813bab5 Nov 03 22:30:06.525150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.525115 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:06.525260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.525184 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:06.525260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.525213 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:06.528467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.528440 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m"] Nov 03 22:30:06.773918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.773810 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7"] Nov 03 22:30:06.778350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.778323 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.837221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.837176 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7"] Nov 03 22:30:06.873929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.873880 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.874129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.873953 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rtz2\" (UniqueName: \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-kube-api-access-6rtz2\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.874129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.874035 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.874129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.874072 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.874129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.874100 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.874344 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.874175 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.874344 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.874308 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.874440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.874369 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.874440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.874401 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.931822 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.931784 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" event={"ID":"05f976bf-42fe-4c21-9408-30fc92f0d5bc","Type":"ContainerStarted","Data":"eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c"} Nov 03 22:30:06.932260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.931848 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" event={"ID":"05f976bf-42fe-4c21-9408-30fc92f0d5bc","Type":"ContainerStarted","Data":"b5f3e3a4118d59983416627bcc91a162f11386090b33b15ad2c8ad639813bab5"} Nov 03 22:30:06.958975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.958937 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:06.975626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.975588 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.975858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.975688 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rtz2\" (UniqueName: \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-kube-api-access-6rtz2\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.975858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.975783 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.975858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.975822 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.975858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.975853 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.976049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.975930 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.976108 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.976055 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.976163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.976116 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.976163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.976148 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.976380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.976355 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.976684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.976632 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.976752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.976718 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.976814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.976765 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.977092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.977060 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.978846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.978824 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:06.979048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:06.979026 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:07.035370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.035301 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rtz2\" (UniqueName: \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-kube-api-access-6rtz2\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:07.035609 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.035587 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:07.058970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.058914 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" podStartSLOduration=2.058897694 podStartE2EDuration="2.058897694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:07.054360643 +0000 UTC m=+1364.035094296" watchObservedRunningTime="2026-11-03 22:30:07.058897694 +0000 UTC m=+1364.039631415" Nov 03 22:30:07.069996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.069963 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:07.076693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.076619 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:07.098087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.098017 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:07.159071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.159035 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:07.369558 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.369473 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp"] Nov 03 22:30:07.375771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.375741 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.450386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.450347 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp"] Nov 03 22:30:07.497161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.497126 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.497393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.497194 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.497393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.497286 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.497393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.497340 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.497393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.497358 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.497622 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.497492 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.497714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.497619 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2411c9-7385-4b74-b850-2006e89a8f54-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.497774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.497707 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znsmv\" (UniqueName: \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-kube-api-access-znsmv\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.497774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.497762 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.519269 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:07.519227 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4e9f4a_6f70_4f72_b2ee_efc803b54cc7.slice/crio-6c86d49bad91bb73df77b506505423dceacc885abddecca41264ca2f77c8acc8 WatchSource:0}: Error finding container 6c86d49bad91bb73df77b506505423dceacc885abddecca41264ca2f77c8acc8: Status 404 returned error can't find the container with id 6c86d49bad91bb73df77b506505423dceacc885abddecca41264ca2f77c8acc8 Nov 03 22:30:07.522623 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.522589 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:07.522803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.522695 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:07.522803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.522732 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:07.523187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.523159 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7"] Nov 03 22:30:07.600062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.600026 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2411c9-7385-4b74-b850-2006e89a8f54-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.600287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.600122 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znsmv\" (UniqueName: \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-kube-api-access-znsmv\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.600287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.600181 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.600287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.600252 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.600528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.600320 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.600528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.600389 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.600528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.600482 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.600528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.600505 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.600788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.600542 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.600892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.600864 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2411c9-7385-4b74-b850-2006e89a8f54-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.601038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.600932 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.601148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.601127 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.601217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.601192 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.601465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.601416 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.602983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.602964 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.604812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.604790 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.660236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.660146 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.660419 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.660362 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znsmv\" (UniqueName: \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-kube-api-access-znsmv\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.693700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.693407 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:07.947663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.947229 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" event={"ID":"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7","Type":"ContainerStarted","Data":"63a92d0dde8f2308ffba3fab046a0ef8af5d35a981109f97b1943149a56ece83"} Nov 03 22:30:07.947663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.947317 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" event={"ID":"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7","Type":"ContainerStarted","Data":"6c86d49bad91bb73df77b506505423dceacc885abddecca41264ca2f77c8acc8"} Nov 03 22:30:07.947663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.947365 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:07.952659 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:07.952342 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:30:08.072420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.072337 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" podStartSLOduration=2.072314973 podStartE2EDuration="2.072314973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:08.068432691 +0000 UTC m=+1365.049166345" watchObservedRunningTime="2026-11-03 22:30:08.072314973 +0000 UTC m=+1365.053048629" Nov 03 22:30:08.088128 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:08.088086 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2411c9_7385_4b74_b850_2006e89a8f54.slice/crio-b19fded1198468c22ab7b5e5493cfed5551db896a612d28b399908bf2d3ed8b8 WatchSource:0}: Error finding container b19fded1198468c22ab7b5e5493cfed5551db896a612d28b399908bf2d3ed8b8: Status 404 returned error can't find the container with id b19fded1198468c22ab7b5e5493cfed5551db896a612d28b399908bf2d3ed8b8 Nov 03 22:30:08.090926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.090891 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:08.091047 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.090974 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:08.091047 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.091019 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:08.091207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.091184 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp"] Nov 03 22:30:08.098173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.098148 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:08.099881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.099857 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:30:08.099989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.099909 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 22:30:08.356722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.356198 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w"] Nov 03 22:30:08.380662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.374929 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.422403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.422360 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w"] Nov 03 22:30:08.518199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.518145 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.518413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.518285 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.518413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.518381 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.518550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.518477 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.518550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.518502 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbjwm\" (UniqueName: \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-kube-api-access-fbjwm\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.518550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.518546 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.518730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.518616 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.518730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.518666 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.518841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.518750 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.620572 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.620487 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.620753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.620615 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.620753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.620724 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.620870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.620823 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.620870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.620851 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbjwm\" (UniqueName: \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-kube-api-access-fbjwm\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.620972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.620896 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.620972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.620965 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.621082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.621000 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.621082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.621011 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.621191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.621106 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.621398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.621363 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.621398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.621390 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.621537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.621426 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.621704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.621678 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.623532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.623499 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.627667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.623758 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.679814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.679778 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.679973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.679867 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbjwm\" (UniqueName: \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-kube-api-access-fbjwm\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:08.700808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.700773 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:09.001693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.997374 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" event={"ID":"4d2411c9-7385-4b74-b850-2006e89a8f54","Type":"ContainerStarted","Data":"81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b"} Nov 03 22:30:09.001693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:08.997442 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" event={"ID":"4d2411c9-7385-4b74-b850-2006e89a8f54","Type":"ContainerStarted","Data":"b19fded1198468c22ab7b5e5493cfed5551db896a612d28b399908bf2d3ed8b8"} Nov 03 22:30:09.110838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.110799 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:30:09.111133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.111105 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 22:30:09.184368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.184290 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podStartSLOduration=2.184266344 podStartE2EDuration="2.184266344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:09.121271 +0000 UTC m=+1366.102004657" watchObservedRunningTime="2026-11-03 22:30:09.184266344 +0000 UTC m=+1366.164999998" Nov 03 22:30:09.203121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.203082 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:09.218013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.217962 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:09.218307 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.218283 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:09.218453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.218436 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:09.256728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.234021 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w"] Nov 03 22:30:09.408632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.408601 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:09.608336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.607320 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:09.694182 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.694150 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:09.695969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.695944 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:09.696110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.695991 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 22:30:09.914365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:09.914287 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:10.039670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:10.039290 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" event={"ID":"d03ed135-ca08-446d-b9ee-f36ece2b49c8","Type":"ContainerStarted","Data":"8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7"} Nov 03 22:30:10.039670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:10.039370 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" event={"ID":"d03ed135-ca08-446d-b9ee-f36ece2b49c8","Type":"ContainerStarted","Data":"d12da4da4cc1edeebe7b2b1daf9fb09554b80da708cced150e1d1517843416cc"} Nov 03 22:30:10.100282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:10.100247 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:30:10.100540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:10.100513 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 22:30:10.154041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:10.152173 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:10.634676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:10.631135 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:10.698670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:10.698076 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:10.698670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:10.698145 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 22:30:10.708667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:10.703009 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:10.708884 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:10.708821 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:10.708956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:10.708878 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:30:11.107668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:11.107246 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:30:11.107668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:11.107314 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 22:30:11.702675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:11.698791 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:11.702675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:11.698862 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 22:30:11.707671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:11.707092 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:11.707671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:11.707152 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:30:11.752667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:11.752106 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:11.959449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:11.958296 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:12.102207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:12.102167 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:30:12.102525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:12.102479 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 22:30:12.170966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:12.170929 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:12.422374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:12.422228 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:12.694738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:12.694560 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:12.694738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:12.694622 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 22:30:12.705678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:12.704973 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:12.705678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:12.705031 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:30:13.128271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.128233 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:30:13.161376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.161333 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 22:30:13.614668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.593110 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podStartSLOduration=5.593086518 podStartE2EDuration="5.593086518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:10.18218886 +0000 UTC m=+1367.162922514" watchObservedRunningTime="2026-11-03 22:30:13.593086518 +0000 UTC m=+1370.573820172" Nov 03 22:30:13.614668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.613154 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5"] Nov 03 22:30:13.628678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.624685 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:13.728657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.727854 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:13.728657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.727918 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:30:13.728657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.728551 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:13.728657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.728590 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 22:30:13.730709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.730445 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:13.765817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.765777 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:13.818676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.806887 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5"] Nov 03 22:30:13.846668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.812535 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:13.872811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.842167 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-socket\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:13.917010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.916975 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-data\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:13.924608 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.924578 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-certs\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:13.925122 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.925095 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2xtj\" (UniqueName: \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-kube-api-access-g2xtj\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:13.926206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.926183 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-credential-socket\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:13.926433 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.926415 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-podinfo\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:13.937078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.937049 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70853f4-2318-4175-9e48-9c8e2b7e0439-istiod-ca-cert\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:13.970099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.970054 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-envoy\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:13.978225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.978187 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-token\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:13.979064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.960993 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:13.983683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:13.971609 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:14.066168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.066135 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-59a07a45-gw-openshift-default-dockercfg-ckmcx\"" Nov 03 22:30:14.124023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.120272 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:30:14.124334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.124304 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 22:30:14.169956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.169897 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:14.214669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.208109 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:14.218334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.218298 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-credential-socket\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.218663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.218626 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-podinfo\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.218865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.218843 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70853f4-2318-4175-9e48-9c8e2b7e0439-istiod-ca-cert\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.219102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.219083 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-envoy\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.219237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.219220 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-token\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.219401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.219383 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-socket\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.219622 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.219604 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-data\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.219763 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.219747 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-certs\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.219902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.219886 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2xtj\" (UniqueName: \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-kube-api-access-g2xtj\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.221816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.221794 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-credential-socket\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.223424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.223360 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70853f4-2318-4175-9e48-9c8e2b7e0439-istiod-ca-cert\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.223851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.223830 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-data\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.224237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.224217 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-socket\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.224674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.224656 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-certs\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.260879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.259658 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-podinfo\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.331529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.331490 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-token\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.332947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.332917 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2xtj\" (UniqueName: \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-kube-api-access-g2xtj\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.364667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.346970 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-envoy\") pod \"e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.364667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.353183 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:14.475675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.474918 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:14.610691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.607024 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:14.699870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.699739 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:14.699870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.699796 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 22:30:14.715993 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.715959 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:14.716255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.716226 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:30:14.923847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.923802 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:14.928939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.928908 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m"] Nov 03 22:30:14.974261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:14.955794 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:15.062668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.037874 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m"] Nov 03 22:30:15.063027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.053096 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:30:15.107848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.107801 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.108138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.108106 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 22:30:15.118927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.118872 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5"] Nov 03 22:30:15.127677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.121799 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fgkt\" (UniqueName: \"kubernetes.io/projected/81cf9ba0-c45b-4fba-92ba-23fba424c30f-kube-api-access-8fgkt\") pod \"maas-api-revoke-sub-3218de71-rlj2m\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:15.128200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.128165 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3218de71-rlj2m\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:15.128412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.128391 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3218de71-rlj2m\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:15.144254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.140795 4298 scope.go:122] "RemoveContainer" containerID="96910ba1b1bcec70b57d823c860f5dff8665be2737a23143212c9778841b13ca" Nov 03 22:30:15.159580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.159550 4298 scope.go:122] "RemoveContainer" containerID="7ef64f42d8c40126fb6c0c9fcaf5f9ede86577523361e004a00374942e76ff6b" Nov 03 22:30:15.213939 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:15.213909 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70853f4_2318_4175_9e48_9c8e2b7e0439.slice/crio-4fe72a6af5d47aa479fa1d95c309da621791289ac93a35ed1947d2302b343018 WatchSource:0}: Error finding container 4fe72a6af5d47aa479fa1d95c309da621791289ac93a35ed1947d2302b343018: Status 404 returned error can't find the container with id 4fe72a6af5d47aa479fa1d95c309da621791289ac93a35ed1947d2302b343018 Nov 03 22:30:15.222788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.222759 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:15.247718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.238352 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3218de71-rlj2m\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:15.247718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.238442 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3218de71-rlj2m\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:15.247718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.238717 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fgkt\" (UniqueName: \"kubernetes.io/projected/81cf9ba0-c45b-4fba-92ba-23fba424c30f-kube-api-access-8fgkt\") pod \"maas-api-revoke-sub-3218de71-rlj2m\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:15.259677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.253778 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3218de71-rlj2m\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:15.278820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.278783 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3218de71-rlj2m\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:15.291419 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.291260 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:15.291761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.291711 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:15.294123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.294093 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:15.301093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.301069 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fgkt\" (UniqueName: \"kubernetes.io/projected/81cf9ba0-c45b-4fba-92ba-23fba424c30f-kube-api-access-8fgkt\") pod \"maas-api-revoke-sub-3218de71-rlj2m\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:15.302037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.301759 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" event={"ID":"d70853f4-2318-4175-9e48-9c8e2b7e0439","Type":"ContainerStarted","Data":"4fe72a6af5d47aa479fa1d95c309da621791289ac93a35ed1947d2302b343018"} Nov 03 22:30:15.421395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.420187 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:15.722119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.721671 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.722119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.721732 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:30:15.722847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.722761 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.722847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:15.722803 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 22:30:16.070715 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:16.065137 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m"] Nov 03 22:30:16.103681 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:16.090227 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cf9ba0_c45b_4fba_92ba_23fba424c30f.slice/crio-8b9c7d2ee4ef8dd0194e25d9be23231dfcb794648916f51cc6430925dd56ed8b WatchSource:0}: Error finding container 8b9c7d2ee4ef8dd0194e25d9be23231dfcb794648916f51cc6430925dd56ed8b: Status 404 returned error can't find the container with id 8b9c7d2ee4ef8dd0194e25d9be23231dfcb794648916f51cc6430925dd56ed8b Nov 03 22:30:16.138491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:16.136981 4298 scope.go:122] "RemoveContainer" containerID="e77f3a4b56bade4037cd07cfe52aac83bb7ba4c5fa7a4d3c07e79fd7f423d0b1" Nov 03 22:30:16.557823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:16.540775 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" event={"ID":"81cf9ba0-c45b-4fba-92ba-23fba424c30f","Type":"ContainerStarted","Data":"8b9c7d2ee4ef8dd0194e25d9be23231dfcb794648916f51cc6430925dd56ed8b"} Nov 03 22:30:16.557823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:16.548388 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" event={"ID":"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4","Type":"ContainerStarted","Data":"bf6e07c46a23812217b4eca10a5b4f5fac2a95f964d97dc5633d850993015b77"} Nov 03 22:30:16.557823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:16.552336 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" event={"ID":"d70853f4-2318-4175-9e48-9c8e2b7e0439","Type":"ContainerStarted","Data":"44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc"} Nov 03 22:30:16.557823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:16.556347 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" event={"ID":"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3","Type":"ContainerStarted","Data":"62085b23b88ecaa6280502ee854f595948f261a76cdd03d9c17406a331b1a673"} Nov 03 22:30:16.557823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:16.556804 4298 scope.go:122] "RemoveContainer" containerID="62085b23b88ecaa6280502ee854f595948f261a76cdd03d9c17406a331b1a673" Nov 03 22:30:16.557823 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:16.557178 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r_odh-ai-gateway-infra(f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" Nov 03 22:30:16.696474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:16.695720 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.696474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:16.695777 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 22:30:16.811254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:16.731128 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.811254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:16.731183 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:30:17.282699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:17.204048 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:17.282699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:17.204105 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:17.282699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:17.204177 4298 scope.go:122] "RemoveContainer" containerID="a56f895e58fb4761d840024a7b130a62668a40c18fe5124de7258c3ee9910e78" Nov 03 22:30:17.408673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:17.295405 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" podStartSLOduration=4.295384775 podStartE2EDuration="4.295384775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:16.97800293 +0000 UTC m=+1373.958736584" watchObservedRunningTime="2026-11-03 22:30:17.295384775 +0000 UTC m=+1374.276118429" Nov 03 22:30:17.490865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:17.459682 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:17.490865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:17.459798 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:17.720739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:17.702223 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:17.720739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:17.702283 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:30:17.971680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:17.956759 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.000950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.000910 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:18.046667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.035000 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:18.114273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.110592 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:30:18.252071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.140503 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.316865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.316126 4298 generic.go:356] "Generic (PLEG): container finished" podID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerID="62085b23b88ecaa6280502ee854f595948f261a76cdd03d9c17406a331b1a673" exitCode=60 Nov 03 22:30:18.317835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.316194 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" event={"ID":"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3","Type":"ContainerDied","Data":"62085b23b88ecaa6280502ee854f595948f261a76cdd03d9c17406a331b1a673"} Nov 03 22:30:18.318415 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.318388 4298 scope.go:122] "RemoveContainer" containerID="7ef64f42d8c40126fb6c0c9fcaf5f9ede86577523361e004a00374942e76ff6b" Nov 03 22:30:18.396489 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.396447 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" event={"ID":"81cf9ba0-c45b-4fba-92ba-23fba424c30f","Type":"ContainerStarted","Data":"d8b50a32e19de10461cff6be75e522e24d9a77301e499b086f90ee69cd335215"} Nov 03 22:30:18.560581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.504104 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.560581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.517450 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:30:18.560581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.537418 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.560581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.538559 4298 scope.go:122] "RemoveContainer" containerID="62085b23b88ecaa6280502ee854f595948f261a76cdd03d9c17406a331b1a673" Nov 03 22:30:18.560581 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:18.539188 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r_odh-ai-gateway-infra(f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" Nov 03 22:30:18.673203 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:18.673166 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:30:18.747108 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.695914 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" event={"ID":"8ca5763c-a422-4d74-8571-1cf7f31a51ff","Type":"ContainerStarted","Data":"fd38bc5d6523f22a6b0966b6c29e66be90a67b69ece29fb9aebab20825920d4f"} Nov 03 22:30:18.844726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.835363 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:18.844726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.835444 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.844726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.841084 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:18.844726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.841160 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:30:18.869021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.868988 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx"] Nov 03 22:30:18.886681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.882247 4298 generic.go:356] "Generic (PLEG): container finished" podID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerID="bf6e07c46a23812217b4eca10a5b4f5fac2a95f964d97dc5633d850993015b77" exitCode=60 Nov 03 22:30:18.887137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.887106 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" event={"ID":"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4","Type":"ContainerDied","Data":"bf6e07c46a23812217b4eca10a5b4f5fac2a95f964d97dc5633d850993015b77"} Nov 03 22:30:18.887333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.887313 4298 scope.go:122] "RemoveContainer" containerID="96910ba1b1bcec70b57d823c860f5dff8665be2737a23143212c9778841b13ca" Nov 03 22:30:18.900004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.893781 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:18.917917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.909384 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:30:18.932710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.919506 4298 scope.go:122] "RemoveContainer" containerID="fd38bc5d6523f22a6b0966b6c29e66be90a67b69ece29fb9aebab20825920d4f" Nov 03 22:30:18.937037 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:18.937000 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7_odh-ai-gateway-infra(8ca5763c-a422-4d74-8571-1cf7f31a51ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" Nov 03 22:30:18.943111 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.943070 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.965347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.958743 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.965347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:18.959847 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:30:19.099881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.099763 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.120833 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.120778 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.188045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.154889 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 22:30:19.534668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.461150 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r"] Nov 03 22:30:19.534668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.528737 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 22:30:19.534668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.529229 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:30:19.583665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.559168 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.798677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.559912 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:30:20.798677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.560056 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.798677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.589754 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.798677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.628789 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:20.798677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.790893 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.798677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.950773 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.798677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.880638 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:20.798677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.989795 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.798677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.131956 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.798677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.132019 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.798677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.169061 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.950940 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.636390 4298 generic.go:356] "Generic (PLEG): container finished" podID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerID="fd38bc5d6523f22a6b0966b6c29e66be90a67b69ece29fb9aebab20825920d4f" exitCode=60 Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:19.923872 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.825617 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.636422 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" event={"ID":"8ca5763c-a422-4d74-8571-1cf7f31a51ff","Type":"ContainerDied","Data":"fd38bc5d6523f22a6b0966b6c29e66be90a67b69ece29fb9aebab20825920d4f"} Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.825946 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.825990 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.826026 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.826061 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.826095 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.826148 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.756375 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.827107 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.827709 4298 scope.go:122] "RemoveContainer" containerID="fd38bc5d6523f22a6b0966b6c29e66be90a67b69ece29fb9aebab20825920d4f" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:20.828042 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7_odh-ai-gateway-infra(8ca5763c-a422-4d74-8571-1cf7f31a51ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.828081 4298 scope.go:122] "RemoveContainer" containerID="e77f3a4b56bade4037cd07cfe52aac83bb7ba4c5fa7a4d3c07e79fd7f423d0b1" Nov 03 22:30:20.869853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:20.551904 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.120174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.119045 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:21.538043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.537990 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:21.800883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.800731 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:30:21.818004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.817967 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.821281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.819571 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.822195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.819606 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:21.822957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.819630 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.835065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.819670 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.835571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.820039 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:30:21.839002 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.820060 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:30:21.908678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.820106 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.908678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.829048 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 22:30:21.908678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.888753 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:21.908678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.907523 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.908678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.888847 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:21.908678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.907816 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.909127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:21.908757 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" podStartSLOduration=7.908740976 podStartE2EDuration="7.908740976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:19.28802493 +0000 UTC m=+1376.268758585" watchObservedRunningTime="2026-11-03 22:30:21.908740976 +0000 UTC m=+1378.889474631" Nov 03 22:30:22.043605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.040990 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:30:22.043605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.041066 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:30:22.043605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.041086 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" event={"ID":"369e758b-9785-4a8e-889c-70003899615f","Type":"ContainerStarted","Data":"61be7270a5d40f45e31c5d0b2c124312bb307d8c1123a7d62148d642f0fb4c7f"} Nov 03 22:30:22.043605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.041600 4298 scope.go:122] "RemoveContainer" containerID="61be7270a5d40f45e31c5d0b2c124312bb307d8c1123a7d62148d642f0fb4c7f" Nov 03 22:30:22.043605 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:22.042056 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk_odh-ai-gateway-infra(369e758b-9785-4a8e-889c-70003899615f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" podUID="369e758b-9785-4a8e-889c-70003899615f" Nov 03 22:30:22.269674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.237491 4298 generic.go:356] "Generic (PLEG): container finished" podID="81cf9ba0-c45b-4fba-92ba-23fba424c30f" containerID="d8b50a32e19de10461cff6be75e522e24d9a77301e499b086f90ee69cd335215" exitCode=0 Nov 03 22:30:22.269674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.238626 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" event={"ID":"81cf9ba0-c45b-4fba-92ba-23fba424c30f","Type":"ContainerDied","Data":"d8b50a32e19de10461cff6be75e522e24d9a77301e499b086f90ee69cd335215"} Nov 03 22:30:22.382138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.382091 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:22.453762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.451327 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" Nov 03 22:30:22.830963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.822773 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:22.830963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.825121 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-kube-api-access-zzmmb\" (UniqueName: \"kubernetes.io/projected/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-kube-api-access-zzmmb\") pod \"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4\" (UID: \"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4\") " Nov 03 22:30:22.830963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.825310 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-maas-api-service-ca\") pod \"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4\" (UID: \"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4\") " Nov 03 22:30:22.836793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.836760 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:23.086465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.902796 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:30:23.087586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.904457 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 22:30:23.087934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.904487 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:30:23.109808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.904523 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:23.138185 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:23.123079 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-maas-api-service-ca" pod "da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" (UID: "da207322-70a5-4bf3-a1ba-40ed3cfaf1d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:23.148281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.904573 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 22:30:23.353220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.919293 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded" Nov 03 22:30:23.353484 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:23.033282 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:23.354309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:22.904542 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 22:30:23.601670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:23.576419 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 22:30:23.768669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:23.661795 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Nov 03 22:30:23.768669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:23.767474 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-kube-api-access-zzmmb" pod "da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" (UID: "da207322-70a5-4bf3-a1ba-40ed3cfaf1d4"). InnerVolumeSpecName "kube-api-access-zzmmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:23.829681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:23.824655 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:30:23.865029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:23.864919 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 22:30:23.866420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:23.866386 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 22:30:23.899660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:23.884583 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Nov 03 22:30:24.063196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:23.982185 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:24.063196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.062691 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:24.063196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.044751 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:24.063196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:23.982495 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:24.063196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.062871 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:24.087945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.087667 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:24.087945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.087851 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:24.088248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.088226 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:24.109064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.108922 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:24.147750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.147634 4298 scope.go:122] "RemoveContainer" containerID="dc2769ccf4577a97ce18c48711ef6cfac792ced5846bce5b69748189433caef5" Nov 03 22:30:24.164934 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:24.161363 4298 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.019s" Nov 03 22:30:24.164934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.161497 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:30:24.164934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.161528 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:24.164934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.161573 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:30:24.178013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.177988 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:30:24.180636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.180615 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:30:24.233667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.206445 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:30:24.251711 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.248878 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7"] Nov 03 22:30:24.277269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.260588 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:30:24.277269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.276181 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:30:24.300076 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.298605 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:24.300076 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.298657 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzmmb\" (UniqueName: \"kubernetes.io/projected/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4-kube-api-access-zzmmb\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:24.306046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.306007 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:24.306330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.306318 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:30:24.351364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.351339 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:30:24.359685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.352489 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:30:24.359860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.359703 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:30:24.403067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.402255 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:30:24.411341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.410827 4298 generic.go:356] "Generic (PLEG): container finished" podID="369e758b-9785-4a8e-889c-70003899615f" containerID="61be7270a5d40f45e31c5d0b2c124312bb307d8c1123a7d62148d642f0fb4c7f" exitCode=60 Nov 03 22:30:24.411341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.411011 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" event={"ID":"369e758b-9785-4a8e-889c-70003899615f","Type":"ContainerDied","Data":"61be7270a5d40f45e31c5d0b2c124312bb307d8c1123a7d62148d642f0fb4c7f"} Nov 03 22:30:24.411341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.411093 4298 scope.go:122] "RemoveContainer" containerID="a56f895e58fb4761d840024a7b130a62668a40c18fe5124de7258c3ee9910e78" Nov 03 22:30:24.411817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.411789 4298 scope.go:122] "RemoveContainer" containerID="61be7270a5d40f45e31c5d0b2c124312bb307d8c1123a7d62148d642f0fb4c7f" Nov 03 22:30:24.413736 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:24.412312 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk_odh-ai-gateway-infra(369e758b-9785-4a8e-889c-70003899615f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" podUID="369e758b-9785-4a8e-889c-70003899615f" Nov 03 22:30:24.415438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.415416 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" Nov 03 22:30:24.415534 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.415468 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx" event={"ID":"da207322-70a5-4bf3-a1ba-40ed3cfaf1d4","Type":"ContainerDied","Data":"9a0cb38b5199883dd666763bb57e20a6aab85710021d013b015939ca572feeaf"} Nov 03 22:30:24.417329 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.417307 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" Nov 03 22:30:24.419946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.419925 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:30:24.421564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.421551 4298 scope.go:122] "RemoveContainer" containerID="f07aa569e6d3b2b9876cf351304737e3ba3da63efd5e1ea62549fecd46570d7a" Nov 03 22:30:24.445142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.444876 4298 scope.go:122] "RemoveContainer" containerID="bf6e07c46a23812217b4eca10a5b4f5fac2a95f964d97dc5633d850993015b77" Nov 03 22:30:24.481788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.481764 4298 scope.go:122] "RemoveContainer" containerID="c1990bd8146a02595bb8bf78c0f5b4f9212e382b233b99b6bc606da959991bbe" Nov 03 22:30:24.511817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.511789 4298 scope.go:122] "RemoveContainer" containerID="aa05e5e4d18f12476680265fd74c35e88628cb6f75d706d0e1612eff196381c1" Nov 03 22:30:24.512849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.512819 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-kube-api-access-z27jx\" (UniqueName: \"kubernetes.io/projected/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-kube-api-access-z27jx\") pod \"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3\" (UID: \"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3\") " Nov 03 22:30:24.513010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.512977 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-maas-api-service-ca\") pod \"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3\" (UID: \"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3\") " Nov 03 22:30:24.514305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.513418 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-maas-api-service-ca" pod "f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" (UID: "f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:24.516305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.515712 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk"] Nov 03 22:30:24.516560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.516492 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:24.516694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.516632 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-kube-api-access-z27jx" pod "f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" (UID: "f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3"). InnerVolumeSpecName "kube-api-access-z27jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:24.543745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.542154 4298 scope.go:122] "RemoveContainer" containerID="185ed392240caa96d2c83bfed586da7f9ddfdcb9bfe371398b8c14af77da8ebd" Nov 03 22:30:24.560268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.560235 4298 scope.go:122] "RemoveContainer" containerID="284389027edc742a1e81518d6a7b6dc7645d0f9801a9b6a113fdd5f5ea5f483f" Nov 03 22:30:24.585528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.585135 4298 scope.go:122] "RemoveContainer" containerID="2e5a07ec6c40f9c8942cb8f8f66022e0fcfe4ebe55f8e651569dbabaaf351ca8" Nov 03 22:30:24.608865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.608835 4298 scope.go:122] "RemoveContainer" containerID="ed3091bc095b580ef16c06935475f047c0a28f7db07abe39b7d723fbc1d7d9a1" Nov 03 22:30:24.618234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.618201 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z27jx\" (UniqueName: \"kubernetes.io/projected/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3-kube-api-access-z27jx\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:24.713050 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.713024 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:24.717498 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.717476 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" Nov 03 22:30:24.775264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.775238 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:24.821180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.821148 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cf9ba0-c45b-4fba-92ba-23fba424c30f-kube-api-access-8fgkt\" (UniqueName: \"kubernetes.io/projected/81cf9ba0-c45b-4fba-92ba-23fba424c30f-kube-api-access-8fgkt\") pod \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " Nov 03 22:30:24.821355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.821211 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-openshift-ca\") pod \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " Nov 03 22:30:24.821355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.821320 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-xks-ca\") pod \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\" (UID: \"81cf9ba0-c45b-4fba-92ba-23fba424c30f\") " Nov 03 22:30:24.821355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.821349 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca5763c-a422-4d74-8571-1cf7f31a51ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca5763c-a422-4d74-8571-1cf7f31a51ff-maas-api-service-ca\") pod \"8ca5763c-a422-4d74-8571-1cf7f31a51ff\" (UID: \"8ca5763c-a422-4d74-8571-1cf7f31a51ff\") " Nov 03 22:30:24.821542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.821476 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca5763c-a422-4d74-8571-1cf7f31a51ff-kube-api-access-gspl8\" (UniqueName: \"kubernetes.io/projected/8ca5763c-a422-4d74-8571-1cf7f31a51ff-kube-api-access-gspl8\") pod \"8ca5763c-a422-4d74-8571-1cf7f31a51ff\" (UID: \"8ca5763c-a422-4d74-8571-1cf7f31a51ff\") " Nov 03 22:30:24.821741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.821717 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-openshift-ca" pod "81cf9ba0-c45b-4fba-92ba-23fba424c30f" (UID: "81cf9ba0-c45b-4fba-92ba-23fba424c30f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:24.821899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.821820 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca5763c-a422-4d74-8571-1cf7f31a51ff-maas-api-service-ca" pod "8ca5763c-a422-4d74-8571-1cf7f31a51ff" (UID: "8ca5763c-a422-4d74-8571-1cf7f31a51ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:24.822212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.822190 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:24.822267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.822222 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca5763c-a422-4d74-8571-1cf7f31a51ff-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:24.823431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.823407 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-xks-ca" pod "81cf9ba0-c45b-4fba-92ba-23fba424c30f" (UID: "81cf9ba0-c45b-4fba-92ba-23fba424c30f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:24.823506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.823440 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cf9ba0-c45b-4fba-92ba-23fba424c30f-kube-api-access-8fgkt" pod "81cf9ba0-c45b-4fba-92ba-23fba424c30f" (UID: "81cf9ba0-c45b-4fba-92ba-23fba424c30f"). InnerVolumeSpecName "kube-api-access-8fgkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:24.823506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.823459 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca5763c-a422-4d74-8571-1cf7f31a51ff-kube-api-access-gspl8" pod "8ca5763c-a422-4d74-8571-1cf7f31a51ff" (UID: "8ca5763c-a422-4d74-8571-1cf7f31a51ff"). InnerVolumeSpecName "kube-api-access-gspl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:24.923953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.923849 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fgkt\" (UniqueName: \"kubernetes.io/projected/81cf9ba0-c45b-4fba-92ba-23fba424c30f-kube-api-access-8fgkt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:24.923953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.923883 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cf9ba0-c45b-4fba-92ba-23fba424c30f-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:24.923953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:24.923898 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gspl8\" (UniqueName: \"kubernetes.io/projected/8ca5763c-a422-4d74-8571-1cf7f31a51ff-kube-api-access-gspl8\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:25.049507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.049466 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:25.120538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.120500 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx"] Nov 03 22:30:25.125177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.125152 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895750-n56hx"] Nov 03 22:30:25.136539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.136512 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" path="/var/lib/kubelet/pods/da207322-70a5-4bf3-a1ba-40ed3cfaf1d4/volumes" Nov 03 22:30:25.423709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.423685 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" Nov 03 22:30:25.423887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.423679 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m" event={"ID":"81cf9ba0-c45b-4fba-92ba-23fba424c30f","Type":"ContainerDied","Data":"8b9c7d2ee4ef8dd0194e25d9be23231dfcb794648916f51cc6430925dd56ed8b"} Nov 03 22:30:25.423887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.423809 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b9c7d2ee4ef8dd0194e25d9be23231dfcb794648916f51cc6430925dd56ed8b" Nov 03 22:30:25.425166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.425146 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" Nov 03 22:30:25.425166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.425159 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7" event={"ID":"8ca5763c-a422-4d74-8571-1cf7f31a51ff","Type":"ContainerDied","Data":"c959ba1a3c070f2b0b92a799e11b83cbd3934e81f497d14adf246fa5df363c5e"} Nov 03 22:30:25.425367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.425214 4298 scope.go:122] "RemoveContainer" containerID="fd38bc5d6523f22a6b0966b6c29e66be90a67b69ece29fb9aebab20825920d4f" Nov 03 22:30:25.429148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.429123 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" Nov 03 22:30:25.429251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.429169 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r" event={"ID":"f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3","Type":"ContainerDied","Data":"f337792f394d3b77b7f71f9ecc0e43adf5d25c3d54a7b9b7f467f59329c06f8d"} Nov 03 22:30:25.430876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.430857 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:30:25.437583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.437564 4298 scope.go:122] "RemoveContainer" containerID="62085b23b88ecaa6280502ee854f595948f261a76cdd03d9c17406a331b1a673" Nov 03 22:30:25.582494 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.582467 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" Nov 03 22:30:25.649422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.649390 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369e758b-9785-4a8e-889c-70003899615f-kube-api-access-smxkk\" (UniqueName: \"kubernetes.io/projected/369e758b-9785-4a8e-889c-70003899615f-kube-api-access-smxkk\") pod \"369e758b-9785-4a8e-889c-70003899615f\" (UID: \"369e758b-9785-4a8e-889c-70003899615f\") " Nov 03 22:30:25.649606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.649594 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369e758b-9785-4a8e-889c-70003899615f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e758b-9785-4a8e-889c-70003899615f-maas-api-service-ca\") pod \"369e758b-9785-4a8e-889c-70003899615f\" (UID: \"369e758b-9785-4a8e-889c-70003899615f\") " Nov 03 22:30:25.650026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.649995 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369e758b-9785-4a8e-889c-70003899615f-maas-api-service-ca" pod "369e758b-9785-4a8e-889c-70003899615f" (UID: "369e758b-9785-4a8e-889c-70003899615f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:25.651528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.651504 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369e758b-9785-4a8e-889c-70003899615f-kube-api-access-smxkk" pod "369e758b-9785-4a8e-889c-70003899615f" (UID: "369e758b-9785-4a8e-889c-70003899615f"). InnerVolumeSpecName "kube-api-access-smxkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:25.751546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.751514 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxkk\" (UniqueName: \"kubernetes.io/projected/369e758b-9785-4a8e-889c-70003899615f-kube-api-access-smxkk\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:25.751546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:25.751543 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e758b-9785-4a8e-889c-70003899615f-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:26.439625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.439597 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" Nov 03 22:30:26.440097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.439628 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk" event={"ID":"369e758b-9785-4a8e-889c-70003899615f","Type":"ContainerDied","Data":"e93e93c42a521ee700a0de9bf621ff9c83a7f5c1e67b2fe1b0cda49540a3e127"} Nov 03 22:30:26.440097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.439706 4298 scope.go:122] "RemoveContainer" containerID="61be7270a5d40f45e31c5d0b2c124312bb307d8c1123a7d62148d642f0fb4c7f" Nov 03 22:30:26.787989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.787957 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7"] Nov 03 22:30:26.792064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.792037 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895750-74tn7"] Nov 03 22:30:26.903097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.903059 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r"] Nov 03 22:30:26.907062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.907040 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895750-scp2r"] Nov 03 22:30:26.945145 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.945113 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.966931 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4"] Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968415 4298 cpu_manager.go:414] "containerMap: removing container" podUID="369e758b-9785-4a8e-889c-70003899615f" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968440 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369e758b-9785-4a8e-889c-70003899615f" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968455 4298 cpu_manager.go:414] "containerMap: removing container" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968462 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968472 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968481 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968506 4298 cpu_manager.go:414] "containerMap: removing container" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968513 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968526 4298 cpu_manager.go:414] "containerMap: removing container" podUID="81cf9ba0-c45b-4fba-92ba-23fba424c30f" containerName="revoke-keys" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968533 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cf9ba0-c45b-4fba-92ba-23fba424c30f" containerName="revoke-keys" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968550 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968558 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968569 4298 cpu_manager.go:414] "containerMap: removing container" podUID="369e758b-9785-4a8e-889c-70003899615f" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968576 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369e758b-9785-4a8e-889c-70003899615f" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968592 4298 cpu_manager.go:414] "containerMap: removing container" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968599 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968611 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968619 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968636 4298 cpu_manager.go:414] "containerMap: removing container" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968659 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968672 4298 cpu_manager.go:414] "containerMap: removing container" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968681 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968697 4298 cpu_manager.go:414] "containerMap: removing container" podUID="369e758b-9785-4a8e-889c-70003899615f" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.968704 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369e758b-9785-4a8e-889c-70003899615f" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.970131 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="369e758b-9785-4a8e-889c-70003899615f" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.970154 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.970165 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.970177 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.970188 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cf9ba0-c45b-4fba-92ba-23fba424c30f" containerName="revoke-keys" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.970198 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.970207 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.970214 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="369e758b-9785-4a8e-889c-70003899615f" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.970223 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.970233 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerName="cleanup" Nov 03 22:30:26.976533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.970242 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" containerName="cleanup" Nov 03 22:30:26.977793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:26.976969 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:27.024229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.024195 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4"] Nov 03 22:30:27.034324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.034287 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:30:27.034760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.034737 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk"] Nov 03 22:30:27.034760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.034745 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:30:27.042038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.041956 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895750-l7lfk"] Nov 03 22:30:27.075463 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.075427 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ebc0ff3-g9wl4\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:27.075687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.075593 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8zt\" (UniqueName: \"kubernetes.io/projected/163fc7a1-7ca2-46ef-9313-208fec197592-kube-api-access-lv8zt\") pod \"maas-api-revoke-sub-5ebc0ff3-g9wl4\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:27.075927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.075900 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ebc0ff3-g9wl4\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:27.136535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.136502 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369e758b-9785-4a8e-889c-70003899615f" path="/var/lib/kubelet/pods/369e758b-9785-4a8e-889c-70003899615f/volumes" Nov 03 22:30:27.137025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.137010 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca5763c-a422-4d74-8571-1cf7f31a51ff" path="/var/lib/kubelet/pods/8ca5763c-a422-4d74-8571-1cf7f31a51ff/volumes" Nov 03 22:30:27.137423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.137411 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" path="/var/lib/kubelet/pods/f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3/volumes" Nov 03 22:30:27.177470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.177427 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ebc0ff3-g9wl4\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:27.177691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.177506 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ebc0ff3-g9wl4\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:27.177691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.177587 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8zt\" (UniqueName: \"kubernetes.io/projected/163fc7a1-7ca2-46ef-9313-208fec197592-kube-api-access-lv8zt\") pod \"maas-api-revoke-sub-5ebc0ff3-g9wl4\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:27.178226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.178200 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ebc0ff3-g9wl4\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:27.179996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.179972 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ebc0ff3-g9wl4\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:27.236595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.236562 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8zt\" (UniqueName: \"kubernetes.io/projected/163fc7a1-7ca2-46ef-9313-208fec197592-kube-api-access-lv8zt\") pod \"maas-api-revoke-sub-5ebc0ff3-g9wl4\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:27.293331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.293253 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:27.324550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.324511 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:27.532416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.525671 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:27.683467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:27.683434 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4"] Nov 03 22:30:27.684021 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:27.683990 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163fc7a1_7ca2_46ef_9313_208fec197592.slice/crio-e217d4387e24bf7c4cb36bdadb17dd1af96c28d6e8ce8374961bdf2c8344caa9 WatchSource:0}: Error finding container e217d4387e24bf7c4cb36bdadb17dd1af96c28d6e8ce8374961bdf2c8344caa9: Status 404 returned error can't find the container with id e217d4387e24bf7c4cb36bdadb17dd1af96c28d6e8ce8374961bdf2c8344caa9 Nov 03 22:30:28.473545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:28.473499 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" event={"ID":"163fc7a1-7ca2-46ef-9313-208fec197592","Type":"ContainerStarted","Data":"f112f0f34be574566f5682a16278cd0ef0c34209ad15283e9da66d86b71dd710"} Nov 03 22:30:28.473742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:28.473557 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" event={"ID":"163fc7a1-7ca2-46ef-9313-208fec197592","Type":"ContainerStarted","Data":"e217d4387e24bf7c4cb36bdadb17dd1af96c28d6e8ce8374961bdf2c8344caa9"} Nov 03 22:30:28.592902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:28.592843 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" podStartSLOduration=2.592828897 podStartE2EDuration="2.592828897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:28.588577828 +0000 UTC m=+1385.569311482" watchObservedRunningTime="2026-11-03 22:30:28.592828897 +0000 UTC m=+1385.573562550" Nov 03 22:30:29.483567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:29.483525 4298 generic.go:356] "Generic (PLEG): container finished" podID="163fc7a1-7ca2-46ef-9313-208fec197592" containerID="f112f0f34be574566f5682a16278cd0ef0c34209ad15283e9da66d86b71dd710" exitCode=0 Nov 03 22:30:29.483750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:29.483607 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" event={"ID":"163fc7a1-7ca2-46ef-9313-208fec197592","Type":"ContainerDied","Data":"f112f0f34be574566f5682a16278cd0ef0c34209ad15283e9da66d86b71dd710"} Nov 03 22:30:30.111976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:30.111935 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:30.294516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:30.294467 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:30.853658 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:30.853283 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:30.961709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:30.961663 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-openshift-ca\") pod \"163fc7a1-7ca2-46ef-9313-208fec197592\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " Nov 03 22:30:30.961915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:30.961894 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163fc7a1-7ca2-46ef-9313-208fec197592-kube-api-access-lv8zt\" (UniqueName: \"kubernetes.io/projected/163fc7a1-7ca2-46ef-9313-208fec197592-kube-api-access-lv8zt\") pod \"163fc7a1-7ca2-46ef-9313-208fec197592\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " Nov 03 22:30:30.962036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:30.962004 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-openshift-ca" pod "163fc7a1-7ca2-46ef-9313-208fec197592" (UID: "163fc7a1-7ca2-46ef-9313-208fec197592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:30.962174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:30.962035 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-xks-ca\") pod \"163fc7a1-7ca2-46ef-9313-208fec197592\" (UID: \"163fc7a1-7ca2-46ef-9313-208fec197592\") " Nov 03 22:30:30.962719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:30.962637 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:30.964115 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:30.964096 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163fc7a1-7ca2-46ef-9313-208fec197592-kube-api-access-lv8zt" pod "163fc7a1-7ca2-46ef-9313-208fec197592" (UID: "163fc7a1-7ca2-46ef-9313-208fec197592"). InnerVolumeSpecName "kube-api-access-lv8zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:30.964253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:30.964236 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-xks-ca" pod "163fc7a1-7ca2-46ef-9313-208fec197592" (UID: "163fc7a1-7ca2-46ef-9313-208fec197592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:31.063904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:31.063813 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv8zt\" (UniqueName: \"kubernetes.io/projected/163fc7a1-7ca2-46ef-9313-208fec197592-kube-api-access-lv8zt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:31.063904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:31.063848 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163fc7a1-7ca2-46ef-9313-208fec197592-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:31.504991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:31.504956 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" Nov 03 22:30:31.504991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:31.504987 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4" event={"ID":"163fc7a1-7ca2-46ef-9313-208fec197592","Type":"ContainerDied","Data":"e217d4387e24bf7c4cb36bdadb17dd1af96c28d6e8ce8374961bdf2c8344caa9"} Nov 03 22:30:31.505398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:31.505037 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e217d4387e24bf7c4cb36bdadb17dd1af96c28d6e8ce8374961bdf2c8344caa9" Nov 03 22:30:32.226602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:32.226557 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:32.750543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:32.750510 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:32.973349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:32.973143 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:34.474351 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:34.474314 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda207322_70a5_4bf3_a1ba_40ed3cfaf1d4.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:37.312894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.312858 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:37.693337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.693261 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r"] Nov 03 22:30:37.693912 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.693896 4298 cpu_manager.go:414] "containerMap: removing container" podUID="163fc7a1-7ca2-46ef-9313-208fec197592" containerName="revoke-keys" Nov 03 22:30:37.693955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.693914 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163fc7a1-7ca2-46ef-9313-208fec197592" containerName="revoke-keys" Nov 03 22:30:37.693999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.693958 4298 cpu_manager.go:414] "containerMap: removing container" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerName="cleanup" Nov 03 22:30:37.693999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.693964 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da207322-70a5-4bf3-a1ba-40ed3cfaf1d4" containerName="cleanup" Nov 03 22:30:37.694062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.694053 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="369e758b-9785-4a8e-889c-70003899615f" containerName="cleanup" Nov 03 22:30:37.694091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.694063 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80b0ef3-a35f-4ea8-b7f2-c70e81a768b3" containerName="cleanup" Nov 03 22:30:37.694091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.694074 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="163fc7a1-7ca2-46ef-9313-208fec197592" containerName="revoke-keys" Nov 03 22:30:37.698912 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.698895 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:37.753678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.753619 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r"] Nov 03 22:30:37.756100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.756058 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6781acfd-jt82r\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:37.756253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.756168 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l7nl\" (UniqueName: \"kubernetes.io/projected/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-kube-api-access-5l7nl\") pod \"maas-api-revoke-sub-6781acfd-jt82r\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:37.756320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.756276 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6781acfd-jt82r\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:37.797176 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.797143 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:30:37.797176 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.797170 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:30:37.858210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.858145 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6781acfd-jt82r\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:37.858414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.858394 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6781acfd-jt82r\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:37.858559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.858544 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l7nl\" (UniqueName: \"kubernetes.io/projected/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-kube-api-access-5l7nl\") pod \"maas-api-revoke-sub-6781acfd-jt82r\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:37.859177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.859154 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6781acfd-jt82r\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:37.860539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.860518 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6781acfd-jt82r\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:37.916915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.916878 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l7nl\" (UniqueName: \"kubernetes.io/projected/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-kube-api-access-5l7nl\") pod \"maas-api-revoke-sub-6781acfd-jt82r\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:37.917519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:37.917493 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:38.014017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:38.013920 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:38.104821 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:38.104774 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:38.454003 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:38.453960 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589c4892_f1f4_4930_b6c0_867f9dbaa5d7.slice/crio-14a03f785d9e345c43e2beee97dc4b2f76319d89bb23098a915e10c6f5f5c95f WatchSource:0}: Error finding container 14a03f785d9e345c43e2beee97dc4b2f76319d89bb23098a915e10c6f5f5c95f: Status 404 returned error can't find the container with id 14a03f785d9e345c43e2beee97dc4b2f76319d89bb23098a915e10c6f5f5c95f Nov 03 22:30:38.460600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:38.460572 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r"] Nov 03 22:30:38.563277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:38.563238 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" event={"ID":"589c4892-f1f4-4930-b6c0-867f9dbaa5d7","Type":"ContainerStarted","Data":"14a03f785d9e345c43e2beee97dc4b2f76319d89bb23098a915e10c6f5f5c95f"} Nov 03 22:30:38.794623 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:38.794582 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda207322_70a5_4bf3_a1ba_40ed3cfaf1d4.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:39.571620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:39.571582 4298 generic.go:356] "Generic (PLEG): container finished" podID="589c4892-f1f4-4930-b6c0-867f9dbaa5d7" containerID="22ca6be03fe88115e499cdbaa0e4b7f97d29a8e6018599c3c880fafb74608300" exitCode=0 Nov 03 22:30:39.572073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:39.571669 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" event={"ID":"589c4892-f1f4-4930-b6c0-867f9dbaa5d7","Type":"ContainerDied","Data":"22ca6be03fe88115e499cdbaa0e4b7f97d29a8e6018599c3c880fafb74608300"} Nov 03 22:30:40.758362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:40.758338 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:40.823378 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:40.823347 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-kube-api-access-5l7nl\" (UniqueName: \"kubernetes.io/projected/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-kube-api-access-5l7nl\") pod \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " Nov 03 22:30:40.823551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:40.823531 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-openshift-ca\") pod \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " Nov 03 22:30:40.823696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:40.823669 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-xks-ca\") pod \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\" (UID: \"589c4892-f1f4-4930-b6c0-867f9dbaa5d7\") " Nov 03 22:30:40.823933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:40.823910 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-openshift-ca" pod "589c4892-f1f4-4930-b6c0-867f9dbaa5d7" (UID: "589c4892-f1f4-4930-b6c0-867f9dbaa5d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:40.824229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:40.824212 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:40.825543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:40.825521 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-kube-api-access-5l7nl" pod "589c4892-f1f4-4930-b6c0-867f9dbaa5d7" (UID: "589c4892-f1f4-4930-b6c0-867f9dbaa5d7"). InnerVolumeSpecName "kube-api-access-5l7nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:40.825594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:40.825543 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-xks-ca" pod "589c4892-f1f4-4930-b6c0-867f9dbaa5d7" (UID: "589c4892-f1f4-4930-b6c0-867f9dbaa5d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:40.925509 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:40.925477 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l7nl\" (UniqueName: \"kubernetes.io/projected/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-kube-api-access-5l7nl\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:40.925509 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:40.925508 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589c4892-f1f4-4930-b6c0-867f9dbaa5d7-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:41.589775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:41.589746 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" Nov 03 22:30:41.589954 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:41.589749 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r" event={"ID":"589c4892-f1f4-4930-b6c0-867f9dbaa5d7","Type":"ContainerDied","Data":"14a03f785d9e345c43e2beee97dc4b2f76319d89bb23098a915e10c6f5f5c95f"} Nov 03 22:30:41.589954 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:41.589861 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14a03f785d9e345c43e2beee97dc4b2f76319d89bb23098a915e10c6f5f5c95f" Nov 03 22:30:42.001432 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:42.001400 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:44.211706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.211661 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:44.552273 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:44.552182 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda207322_70a5_4bf3_a1ba_40ed3cfaf1d4.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:44.705846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.705804 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv"] Nov 03 22:30:44.706888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.706864 4298 cpu_manager.go:414] "containerMap: removing container" podUID="589c4892-f1f4-4930-b6c0-867f9dbaa5d7" containerName="revoke-keys" Nov 03 22:30:44.706888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.706890 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589c4892-f1f4-4930-b6c0-867f9dbaa5d7" containerName="revoke-keys" Nov 03 22:30:44.707087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.707044 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="589c4892-f1f4-4930-b6c0-867f9dbaa5d7" containerName="revoke-keys" Nov 03 22:30:44.712449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.712429 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:44.939737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.930527 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-credential-socket\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:44.941747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.940035 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-data\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:44.948612 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.948586 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-certs\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:44.949003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.948982 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xz2\" (UniqueName: \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-kube-api-access-86xz2\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:44.950553 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.949241 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-podinfo\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:44.950937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.950909 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-socket\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:44.951138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.951123 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-envoy\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:44.951265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.951251 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221e35d0-71d1-4488-8108-6dcb28e6b621-istiod-ca-cert\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:44.951390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.951375 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-token\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:44.951664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.942219 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:44.951906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:44.943014 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e1c72fb7-openshift-default-dockercfg-rm7j4\"" Nov 03 22:30:45.014057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.014018 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv"] Nov 03 22:30:45.053416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.053381 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86xz2\" (UniqueName: \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-kube-api-access-86xz2\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.053628 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.053507 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-podinfo\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.053628 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.053599 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-socket\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.053753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.053655 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-envoy\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.053753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.053674 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221e35d0-71d1-4488-8108-6dcb28e6b621-istiod-ca-cert\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.053753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.053690 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-token\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.053753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.053720 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-credential-socket\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.053753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.053747 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-data\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.053914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.053777 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-certs\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.054328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.054304 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-credential-socket\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.054505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.054489 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-data\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.054610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.054582 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221e35d0-71d1-4488-8108-6dcb28e6b621-istiod-ca-cert\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.054698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.054680 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-socket\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.054966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.054946 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-certs\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.058134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.058108 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-podinfo\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.058967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.058931 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-envoy\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.113229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.113193 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-token\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.113416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.113346 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xz2\" (UniqueName: \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-kube-api-access-86xz2\") pod \"e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.326431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.326395 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:45.491713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.491175 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:45.757043 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:45.757012 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221e35d0_71d1_4488_8108_6dcb28e6b621.slice/crio-366501e8784016bc0332d748c0d047b85430167edce8a9d19df54dcfb8c6fb97 WatchSource:0}: Error finding container 366501e8784016bc0332d748c0d047b85430167edce8a9d19df54dcfb8c6fb97: Status 404 returned error can't find the container with id 366501e8784016bc0332d748c0d047b85430167edce8a9d19df54dcfb8c6fb97 Nov 03 22:30:45.759891 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.759853 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:45.759991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.759936 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:45.759991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.759980 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:30:45.761969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:45.761947 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv"] Nov 03 22:30:46.563702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:46.562036 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:46.630950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:46.630904 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" event={"ID":"221e35d0-71d1-4488-8108-6dcb28e6b621","Type":"ContainerStarted","Data":"b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3"} Nov 03 22:30:46.631147 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:46.631006 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" event={"ID":"221e35d0-71d1-4488-8108-6dcb28e6b621","Type":"ContainerStarted","Data":"366501e8784016bc0332d748c0d047b85430167edce8a9d19df54dcfb8c6fb97"} Nov 03 22:30:46.751936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:46.751878 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" podStartSLOduration=2.751855202 podStartE2EDuration="2.751855202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:46.751135839 +0000 UTC m=+1403.731869492" watchObservedRunningTime="2026-11-03 22:30:46.751855202 +0000 UTC m=+1403.732588849" Nov 03 22:30:47.326812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:47.326778 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:47.332730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:47.332701 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:47.554233 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:47.554190 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:30:47.638834 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:47.638751 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:47.639881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:47.639857 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:30:49.625293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:49.625263 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:49.824961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:49.824927 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:50.761277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:50.761246 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:50.946339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:50.946305 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:51.620132 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:51.620094 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:54.267521 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:54.267482 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda207322_70a5_4bf3_a1ba_40ed3cfaf1d4.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:54.612726 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:30:54.599187 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda207322_70a5_4bf3_a1ba_40ed3cfaf1d4.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:54.811832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:54.811776 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:54.912533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:54.912420 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:55.097668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:55.096799 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:55.101669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:55.098106 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:30:55.118328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:55.118276 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:55.218203 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:55.188747 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:55.364799 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:55.364732 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:30:55.365828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:55.365358 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:55.918810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:55.918754 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 22:30:55.920077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:55.920034 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:30:55.922496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:55.920326 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:30:56.053671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:56.048882 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:56.111850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:56.109838 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:30:56.111850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:56.110173 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:30:56.112174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:56.112146 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:56.130679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:56.112494 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:30:56.202726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:56.193029 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:56.202726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:56.193398 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:30:56.306888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:56.306857 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:30:56.307161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:56.307145 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:30:56.421259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:56.413362 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:30:56.549116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:56.548864 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:30:57.745956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:57.745927 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:57.983564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:57.983517 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:58.828533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:58.828502 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:59.016998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.016955 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:59.214194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.214155 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4"] Nov 03 22:30:59.222666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.221964 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:30:59.307492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.307458 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c497ebcf-a920-40cd-a477-38ac0eeb6350-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4\" (UID: \"c497ebcf-a920-40cd-a477-38ac0eeb6350\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:30:59.308382 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.308355 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frl8v\" (UniqueName: \"kubernetes.io/projected/c497ebcf-a920-40cd-a477-38ac0eeb6350-kube-api-access-frl8v\") pod \"payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4\" (UID: \"c497ebcf-a920-40cd-a477-38ac0eeb6350\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:30:59.318499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.318471 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e1c72fb7\"" Nov 03 22:30:59.322904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.322870 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4"] Nov 03 22:30:59.332846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.332822 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e1c72fb7-dockercfg-x9gxd\"" Nov 03 22:30:59.410196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.410158 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c497ebcf-a920-40cd-a477-38ac0eeb6350-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4\" (UID: \"c497ebcf-a920-40cd-a477-38ac0eeb6350\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:30:59.410387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.410250 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frl8v\" (UniqueName: \"kubernetes.io/projected/c497ebcf-a920-40cd-a477-38ac0eeb6350-kube-api-access-frl8v\") pod \"payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4\" (UID: \"c497ebcf-a920-40cd-a477-38ac0eeb6350\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:30:59.411004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.410986 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c497ebcf-a920-40cd-a477-38ac0eeb6350-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4\" (UID: \"c497ebcf-a920-40cd-a477-38ac0eeb6350\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:30:59.510726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.509985 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frl8v\" (UniqueName: \"kubernetes.io/projected/c497ebcf-a920-40cd-a477-38ac0eeb6350-kube-api-access-frl8v\") pod \"payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4\" (UID: \"c497ebcf-a920-40cd-a477-38ac0eeb6350\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:30:59.553561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:30:59.553266 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:30:59.993152 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:30:59.993117 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc497ebcf_a920_40cd_a477_38ac0eeb6350.slice/crio-b9d2bbd3db7230bbf05cd911026c48aadda68a0362009267fdfb0e384450557a WatchSource:0}: Error finding container b9d2bbd3db7230bbf05cd911026c48aadda68a0362009267fdfb0e384450557a: Status 404 returned error can't find the container with id b9d2bbd3db7230bbf05cd911026c48aadda68a0362009267fdfb0e384450557a Nov 03 22:31:00.000300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:00.000267 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4"] Nov 03 22:31:00.810010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:00.809971 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" event={"ID":"c497ebcf-a920-40cd-a477-38ac0eeb6350","Type":"ContainerStarted","Data":"a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932"} Nov 03 22:31:00.810232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:00.810033 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" event={"ID":"c497ebcf-a920-40cd-a477-38ac0eeb6350","Type":"ContainerStarted","Data":"b9d2bbd3db7230bbf05cd911026c48aadda68a0362009267fdfb0e384450557a"} Nov 03 22:31:00.810232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:00.810062 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:31:00.928928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:00.928875 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" podStartSLOduration=1.9288608630000001 podStartE2EDuration="1.928860863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:00.926097278 +0000 UTC m=+1417.906830931" watchObservedRunningTime="2026-11-03 22:31:00.928860863 +0000 UTC m=+1417.909594515" Nov 03 22:31:02.909072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:02.909018 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:03.115356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:03.115319 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:03.332615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:03.332566 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:04.703033 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:31:04.703002 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda207322_70a5_4bf3_a1ba_40ed3cfaf1d4.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:31:07.227619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:07.227581 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:08.040340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:08.040304 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:08.219958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:08.219922 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:08.499291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:08.499258 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:08.801218 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:31:08.801127 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda207322_70a5_4bf3_a1ba_40ed3cfaf1d4.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:31:11.040599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:11.040566 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:11.821687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:11.821635 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:31:12.754468 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:31:12.754435 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda207322_70a5_4bf3_a1ba_40ed3cfaf1d4.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:31:12.754982 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:31:12.754435 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda207322_70a5_4bf3_a1ba_40ed3cfaf1d4.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:31:12.913052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:12.913014 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:13.567841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:13.567775 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:14.717142 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:31:14.717105 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda207322_70a5_4bf3_a1ba_40ed3cfaf1d4.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:31:17.553888 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:31:17.553847 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:31:17.564655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:17.564609 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:17.749906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:17.749876 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:17.932985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:17.932896 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:21.204772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:21.204739 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:21.389564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:21.389527 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:22.203044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:22.203012 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:23.239325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.239277 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld"] Nov 03 22:31:23.243897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.243877 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.290400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.290365 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww"] Nov 03 22:31:23.294780 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.294757 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.301678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.301630 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.301821 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.301795 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a3eb67-b858-4181-b161-8204932fe1a6-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.301893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.301857 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.302071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.301958 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.302071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.302011 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.302176 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.302119 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvfln\" (UniqueName: \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-kube-api-access-vvfln\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.302244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.302192 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a3eb67-b858-4181-b161-8204932fe1a6-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.302287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.302250 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.302352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.302339 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.341568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.341526 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld"] Nov 03 22:31:23.391808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.391775 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww"] Nov 03 22:31:23.403826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.403793 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.403826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.403831 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.404069 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.403934 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7grm\" (UniqueName: \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-kube-api-access-d7grm\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.404069 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.403960 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.404185 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404080 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.404185 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404174 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.404313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404220 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.404313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404241 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a3eb67-b858-4181-b161-8204932fe1a6-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.404538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404508 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.404538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404527 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.404759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404546 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.404759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404552 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.404759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404747 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.404938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404790 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.404938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404823 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.404938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.404842 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.405098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.405084 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.405274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.405246 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.405341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.405298 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.405395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.405336 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfln\" (UniqueName: \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-kube-api-access-vvfln\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.405710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.405686 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a3eb67-b858-4181-b161-8204932fe1a6-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.405802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.405734 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.406473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.406450 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a3eb67-b858-4181-b161-8204932fe1a6-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.407452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.407426 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a3eb67-b858-4181-b161-8204932fe1a6-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.408534 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.408513 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.465765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.465732 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.465958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.465845 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfln\" (UniqueName: \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-kube-api-access-vvfln\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.507606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.507518 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.507606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.507598 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.507855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.507627 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.507855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.507716 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.507855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.507747 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.507855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.507798 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7grm\" (UniqueName: \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-kube-api-access-d7grm\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.507855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.507824 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.508121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.508014 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.508121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.508075 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.508227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.508201 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.508356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.508322 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.508568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.508415 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.508620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.508565 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.508620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.508607 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.510085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.510066 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.510204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.510187 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.559121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.559082 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:23.566229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.566196 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.566398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.566379 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7grm\" (UniqueName: \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-kube-api-access-d7grm\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.608817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.608783 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:23.639959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.635897 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz"] Nov 03 22:31:23.646885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.646425 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.651351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.649450 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v"] Nov 03 22:31:23.656365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.654403 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.698190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.698142 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz"] Nov 03 22:31:23.711204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.710707 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.711204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.710758 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.711204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.710799 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.711204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.710824 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1728fe75-73a1-4f80-82d6-58b961ed5487-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.711204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.710869 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.711204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711004 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.711204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711051 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.711204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711081 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.711204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711126 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.711204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711182 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45bw\" (UniqueName: \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-kube-api-access-g45bw\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.711845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711219 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.711845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711304 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.711845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711334 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.711845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711402 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.711845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711443 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrg78\" (UniqueName: \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-kube-api-access-wrg78\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.711845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711530 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.711845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711565 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.711845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.711601 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.715038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.714986 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v"] Nov 03 22:31:23.812626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.812524 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.812626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.812564 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.812626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.812592 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.812626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.812610 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1728fe75-73a1-4f80-82d6-58b961ed5487-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.813021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.812654 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.813021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.812766 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.813021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.812810 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.813021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.812835 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.813021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.812885 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.813021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.812925 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45bw\" (UniqueName: \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-kube-api-access-g45bw\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.813021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.812961 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.813416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.813044 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.813416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.813075 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.813416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.813142 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.813416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.813151 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.813416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.813183 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrg78\" (UniqueName: \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-kube-api-access-wrg78\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.813416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.813234 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.813416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.813258 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.813416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.813310 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.813885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.813467 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1728fe75-73a1-4f80-82d6-58b961ed5487-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.813885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.813855 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.814116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.813929 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.814116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.814042 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.814230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.814153 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.814324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.814303 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.814827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.814404 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.814827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.814544 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.815350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.815330 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.815529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.815498 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.815795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.815772 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.816632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.816610 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.816988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.816965 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.871635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.871594 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.872025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.872002 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrg78\" (UniqueName: \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-kube-api-access-wrg78\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.873476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.873447 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45bw\" (UniqueName: \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-kube-api-access-g45bw\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.873632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.873612 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.970010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.969973 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:23.981613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.981583 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:23.987810 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:31:23.986936 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a3eb67_b858_4181_b161_8204932fe1a6.slice/crio-b6520d6296e1c9cb7a4a59adebc063b7b1efdea5d769abf32c35b40403e4c2bd WatchSource:0}: Error finding container b6520d6296e1c9cb7a4a59adebc063b7b1efdea5d769abf32c35b40403e4c2bd: Status 404 returned error can't find the container with id b6520d6296e1c9cb7a4a59adebc063b7b1efdea5d769abf32c35b40403e4c2bd Nov 03 22:31:23.990744 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.990717 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld"] Nov 03 22:31:23.993298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.993246 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:23.993425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.993333 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:23.993425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:23.993376 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:24.015068 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.014914 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" event={"ID":"19a3eb67-b858-4181-b161-8204932fe1a6","Type":"ContainerStarted","Data":"b6520d6296e1c9cb7a4a59adebc063b7b1efdea5d769abf32c35b40403e4c2bd"} Nov 03 22:31:24.045915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.045780 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww"] Nov 03 22:31:24.063259 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:31:24.061810 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98055b9b_6d24_4b65_8e53_d0279f4f6fb0.slice/crio-08ce95015c192c135f7b6428cd0a5f16ee1b43d9b53ae91857cf799c1209032f WatchSource:0}: Error finding container 08ce95015c192c135f7b6428cd0a5f16ee1b43d9b53ae91857cf799c1209032f: Status 404 returned error can't find the container with id 08ce95015c192c135f7b6428cd0a5f16ee1b43d9b53ae91857cf799c1209032f Nov 03 22:31:24.066622 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.066577 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:24.066737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.066693 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:24.066810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.066742 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:24.357060 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:31:24.357024 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1728fe75_73a1_4f80_82d6_58b961ed5487.slice/crio-71765520efaf2ed45d04eca73de11b6914374e20c14d138f2f975642e33bd9c7 WatchSource:0}: Error finding container 71765520efaf2ed45d04eca73de11b6914374e20c14d138f2f975642e33bd9c7: Status 404 returned error can't find the container with id 71765520efaf2ed45d04eca73de11b6914374e20c14d138f2f975642e33bd9c7 Nov 03 22:31:24.360681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.360623 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:24.360791 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.360727 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:24.360791 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.360770 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:24.364347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.362192 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz"] Nov 03 22:31:24.380292 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:31:24.380254 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4986afe5_cdd2_45d7_8012_bcbe881af8f3.slice/crio-cc41f9654cd05f7b5c487c90ce7e101bea382b97b3d53a68e704ea6f3c60f949 WatchSource:0}: Error finding container cc41f9654cd05f7b5c487c90ce7e101bea382b97b3d53a68e704ea6f3c60f949: Status 404 returned error can't find the container with id cc41f9654cd05f7b5c487c90ce7e101bea382b97b3d53a68e704ea6f3c60f949 Nov 03 22:31:24.383603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.383562 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:24.383723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.383675 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:24.383782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.383730 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:24.384769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:24.384747 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v"] Nov 03 22:31:25.078777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.076167 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" event={"ID":"98055b9b-6d24-4b65-8e53-d0279f4f6fb0","Type":"ContainerStarted","Data":"890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a"} Nov 03 22:31:25.078777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.076260 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" event={"ID":"98055b9b-6d24-4b65-8e53-d0279f4f6fb0","Type":"ContainerStarted","Data":"08ce95015c192c135f7b6428cd0a5f16ee1b43d9b53ae91857cf799c1209032f"} Nov 03 22:31:25.098668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.080280 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" event={"ID":"1728fe75-73a1-4f80-82d6-58b961ed5487","Type":"ContainerStarted","Data":"2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9"} Nov 03 22:31:25.098668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.080354 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" event={"ID":"1728fe75-73a1-4f80-82d6-58b961ed5487","Type":"ContainerStarted","Data":"71765520efaf2ed45d04eca73de11b6914374e20c14d138f2f975642e33bd9c7"} Nov 03 22:31:25.098668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.085429 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" event={"ID":"19a3eb67-b858-4181-b161-8204932fe1a6","Type":"ContainerStarted","Data":"d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e"} Nov 03 22:31:25.098668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.089200 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" event={"ID":"4986afe5-cdd2-45d7-8012-bcbe881af8f3","Type":"ContainerStarted","Data":"062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575"} Nov 03 22:31:25.098668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.089262 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" event={"ID":"4986afe5-cdd2-45d7-8012-bcbe881af8f3","Type":"ContainerStarted","Data":"cc41f9654cd05f7b5c487c90ce7e101bea382b97b3d53a68e704ea6f3c60f949"} Nov 03 22:31:25.247666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.246916 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podStartSLOduration=2.246897075 podStartE2EDuration="2.246897075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:25.246151369 +0000 UTC m=+1442.226885056" watchObservedRunningTime="2026-11-03 22:31:25.246897075 +0000 UTC m=+1442.227630729" Nov 03 22:31:25.455669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.450313 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podStartSLOduration=2.4502903050000002 podStartE2EDuration="2.450290305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:25.368910799 +0000 UTC m=+1442.349644470" watchObservedRunningTime="2026-11-03 22:31:25.450290305 +0000 UTC m=+1442.431023959" Nov 03 22:31:25.531670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.528245 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podStartSLOduration=2.52822038 podStartE2EDuration="2.52822038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:25.501155256 +0000 UTC m=+1442.481888944" watchObservedRunningTime="2026-11-03 22:31:25.52822038 +0000 UTC m=+1442.508954035" Nov 03 22:31:25.564695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.561016 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:25.573099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.572856 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:31:25.573099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.572918 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 22:31:25.618669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.616795 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:25.618669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.616941 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:31:25.618669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.616986 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 22:31:25.657668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.656619 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podStartSLOduration=2.656594718 podStartE2EDuration="2.656594718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:25.629774669 +0000 UTC m=+1442.610508322" watchObservedRunningTime="2026-11-03 22:31:25.656594718 +0000 UTC m=+1442.637328372" Nov 03 22:31:25.974667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.974467 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:25.974667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.974600 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:31:25.974667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.974660 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 22:31:25.997933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.997881 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:25.998395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.998366 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:31:25.998623 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:25.998597 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 22:31:26.375726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.372886 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:26.390667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.386852 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:26.429666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.418957 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:26.576740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.572783 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:31:26.576740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.572847 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 22:31:26.740761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.641042 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:31:26.740761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.641101 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 22:31:26.814754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.814704 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:26.856100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.855752 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:26.941674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.930869 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:26.941674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.931505 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:26.980685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.980077 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:31:26.980685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.980138 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 22:31:26.992178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.992089 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:31:26.992808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:26.992772 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 22:31:27.337759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.321758 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:27.383062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.374785 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 22:31:27.383062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.374970 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:27.386753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.384047 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:31:27.452668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.401462 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:27.452668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.437754 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:27.452668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.438280 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:31:27.574507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.574111 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:27.601233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.601126 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:31:27.689667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.680402 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:27.748811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.725391 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:31:27.748811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.725473 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 22:31:27.771840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.753098 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 22:31:27.871749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.826922 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:31:27.871749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.867369 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:31:27.871749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.868122 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:27.871749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.869512 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:31:27.953668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.952731 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:27.953668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.953144 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:31:27.953668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.953518 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:31:27.953668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.953600 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:31:28.048669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.991463 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:31:28.048669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:27.991528 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 22:31:28.080667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.050407 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:31:28.253754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.253706 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 22:31:28.274722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.253580 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 22:31:28.358043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.340722 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:28.375153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.369860 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:31:28.375153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.369920 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:31:28.375153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.369959 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:31:28.375153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.369993 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:31:28.375153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.370026 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:31:28.398714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.386741 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:28.398714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.388148 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:31:28.445561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.445516 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:28.574741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.542169 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:31:28.605778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.605741 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:31:28.607403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.607366 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 22:31:28.619754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.611985 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:31:28.620000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.619968 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 22:31:28.902727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.882185 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:28.958672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.955761 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:28.980720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.979903 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:31:28.980720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.979961 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 22:31:28.999980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:28.999940 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:31:29.009821 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.009770 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 22:31:29.320668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.320035 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:29.320668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.320510 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:31:29.391680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.388774 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:29.391680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.389196 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:31:29.394678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.393306 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:31:29.465833 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.465787 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:29.466481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.466457 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:31:29.622655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.591783 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:31:29.622655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.591871 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 22:31:29.622655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.602091 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 22:31:29.678731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.630998 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:29.678731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.631776 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:31:29.678731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.631816 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 22:31:29.895720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:29.885787 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 22:31:30.079670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.065779 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:30.153728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.137177 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:31:30.153728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.137243 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 22:31:30.153728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.139164 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:31:30.153728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.139211 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 22:31:30.153728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.140942 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:31:30.191186 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.191148 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:30.469801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.469351 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:30.491537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.470933 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:30.491537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.471129 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:30.491537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.471327 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:31:30.491537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.471379 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:31:30.491537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.472407 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:30.491537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.473747 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:30.577862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.577808 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:31:30.578246 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.578131 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 22:31:30.653490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.643800 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:31:30.653490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.643862 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 22:31:30.653490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.643928 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 22:31:30.653490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.652809 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:31:30.666008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.655100 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:30.688798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.666348 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:31:30.900137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:30.900037 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:31.047475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.047431 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:31:31.047947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.047824 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 22:31:31.049497 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.049472 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:31:31.049764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.049740 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 22:31:31.233667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.223025 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:31.352671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.349802 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 22:31:31.400846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.400797 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:31.484727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.483741 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:31.484727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.484251 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:31:31.485145 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.485114 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:31.485612 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.485586 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:31:31.581988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.581940 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:31:31.582491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.582453 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 22:31:31.624219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.624176 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:31:31.643802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.643756 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 22:31:31.671670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.646078 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:31.751764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.672391 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:31.926668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.901767 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 22:31:31.973665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.972136 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:31:31.973665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.972206 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 22:31:31.993669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.984265 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:31:31.993669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:31.984334 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 22:31:32.057194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.057089 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:31:32.235800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.235746 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:32.383727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.364773 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:32.383727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.365245 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:31:32.404587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.402356 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:32.418354 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.404957 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 22:31:32.418354 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.405597 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:31:32.507755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.502157 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:31:32.507984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.507758 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:32.509080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.509044 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:32.509221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.509140 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:31:32.510019 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.509997 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:31:32.622717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.619890 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:31:32.622717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.619946 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 22:31:32.638721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.629126 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:31:32.638721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.629187 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 22:31:32.663664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.661874 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded" Nov 03 22:31:32.773672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.771829 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:32.782679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.781086 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:32.903565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:32.903153 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:33.039822 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.039777 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:31:33.040146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.040116 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 22:31:33.043665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.040204 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:31:33.043929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.043894 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 22:31:33.505706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.403566 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:33.505706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.417414 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:33.505706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.417513 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:33.505706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.417557 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:33.549712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.548745 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:31:33.549712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.549087 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:33.587794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.585711 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:31:33.587794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.585771 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 22:31:33.630749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.599504 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:31:33.630749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.630216 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:31:33.630749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.630296 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:31:33.668737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.661092 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:31:33.668737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.661157 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 22:31:33.668737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.662652 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:33.806151 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.806030 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 22:31:33.856695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.821999 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 22:31:33.917668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.915811 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Nov 03 22:31:33.963671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.938963 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:34.063723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:33.969293 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:34.063723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.012081 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:31:34.063723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.038072 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 22:31:34.063723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.014410 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:31:34.063723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.038130 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 22:31:34.074138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.074098 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:34.123965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.119738 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:34.295712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.295681 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:31:34.563794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.563744 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 22:31:34.593097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.593054 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:31:34.626545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.598932 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:34.626545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.599732 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:31:34.626545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.600039 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:31:34.626545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.601536 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:31:34.674665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.671627 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:34.674972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.674945 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:31:34.675109 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.675089 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 22:31:34.679666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.675808 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:31:34.679666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.675862 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 22:31:34.849030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.848898 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:34.972389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.972349 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Nov 03 22:31:34.972674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:34.921943 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:35.039894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.034725 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:35.041878 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.041848 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:31:35.074401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.074355 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 22:31:35.079828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.079782 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:35.084687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.082346 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:31:35.084687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.082398 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 22:31:35.094346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.094315 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:31:35.574981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.574905 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:35.618721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.604005 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:35.620026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.619994 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:31:35.629687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.620221 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 22:31:35.690675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.676499 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:31:35.690675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.676590 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:31:35.690675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.676668 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:31:35.690675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.676733 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:31:35.690675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.676788 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:31:35.820920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.820876 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:31:35.843725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.834826 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:31:35.907631 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.907600 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:31:35.930102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.929721 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:31:35.935098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:35.935076 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:31:36.013671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.001275 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:36.013671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.009582 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:31:36.013671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.009635 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 22:31:36.013671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.012934 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:31:36.013671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.012977 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 22:31:36.121805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.121337 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:36.123335 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.123236 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:36.131785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.131728 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:31:36.135044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.134153 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:31:36.851665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.844945 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:36.851665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.849370 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:36.856584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:36.856265 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:31:37.088668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:37.087884 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:38.013802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.013764 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:31:38.014001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.013858 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:38.023883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.023839 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:31:38.024081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.023910 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:38.553230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.550434 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:38.553230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.552130 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:38.554319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.554292 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:31:38.692230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.692195 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:38.692544 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.692523 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:38.699873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.699842 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:31:38.995862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.995820 4298 patch_prober.go:29] interesting pod/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:31:38.996072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:38.995909 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:40.709289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:40.709248 4298 generic.go:356] "Generic (PLEG): container finished" podID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerID="eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c" exitCode=0 Nov 03 22:31:40.709774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:40.709288 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" event={"ID":"05f976bf-42fe-4c21-9408-30fc92f0d5bc","Type":"ContainerDied","Data":"eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c"} Nov 03 22:31:40.709878 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:40.709772 4298 scope.go:122] "RemoveContainer" containerID="eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c" Nov 03 22:31:40.711391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:40.711360 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:40.711497 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:40.711448 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:40.713379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:40.711887 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:31:41.688586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:41.688546 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:41.738694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:41.737087 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" event={"ID":"05f976bf-42fe-4c21-9408-30fc92f0d5bc","Type":"ContainerStarted","Data":"3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9"} Nov 03 22:31:42.069561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:42.069530 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:31:42.069827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:42.069802 4298 patch_prober.go:29] interesting pod/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 22:31:42.069916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:42.069861 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 22:31:42.977330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:42.977298 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:43.069725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:43.069691 4298 patch_prober.go:29] interesting pod/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 22:31:43.069901 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:43.069742 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 22:31:44.069909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:44.069858 4298 patch_prober.go:29] interesting pod/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 22:31:44.070350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:44.069922 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 22:31:45.069506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:45.069454 4298 patch_prober.go:29] interesting pod/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 22:31:45.069730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:45.069517 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 22:31:46.069728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:46.069691 4298 patch_prober.go:29] interesting pod/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 22:31:46.070147 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:46.069743 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 22:31:46.190121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:46.190090 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:47.070136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:47.070097 4298 patch_prober.go:29] interesting pod/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 22:31:47.070557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:47.070154 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 22:31:47.553633 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:31:47.553592 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:31:48.073905 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:48.073872 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:31:48.074389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:48.074350 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:31:48.075153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:48.075131 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:31:48.761127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:48.761077 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:50.738988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:50.738942 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:52.977059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:52.977024 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:55.484932 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:55.484890 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:58.604586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:58.604538 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:58.783259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:31:58.783209 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:00.080096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:00.080050 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:04.887789 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:04.887691 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:10.205559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:10.205519 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:11.313187 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:32:11.313136 4298 controller.go:251] "Failed to update lease" err="Put \"https://a3890ba9e34074c9bb3a6df7076aca19-87013095a60c6b5c.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-254.us-west-2.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:11.377531 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:11.377492 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:15.573621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:15.573581 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:17.554979 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:32:17.554934 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:32:18.680900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:18.680850 4298 patch_prober.go:29] interesting pod/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:32:18.681390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:18.680931 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:19.251196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:19.237850 4298 patch_prober.go:29] interesting pod/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": read tcp 10.133.0.2:41512->10.133.0.114:15021: read: connection reset by peer" start-of-body= Nov 03 22:32:19.251196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:19.237932 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": read tcp 10.133.0.2:41512->10.133.0.114:15021: read: connection reset by peer" Nov 03 22:32:20.234344 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.234311 4298 generic.go:356] "Generic (PLEG): container finished" podID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerID="44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc" exitCode=0 Nov 03 22:32:20.234802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.234392 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" event={"ID":"d70853f4-2318-4175-9e48-9c8e2b7e0439","Type":"ContainerDied","Data":"44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc"} Nov 03 22:32:20.234985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.234966 4298 scope.go:122] "RemoveContainer" containerID="44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc" Nov 03 22:32:20.236256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.236231 4298 generic.go:356] "Generic (PLEG): container finished" podID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerID="b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3" exitCode=0 Nov 03 22:32:20.236399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.236315 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" event={"ID":"221e35d0-71d1-4488-8108-6dcb28e6b621","Type":"ContainerDied","Data":"b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3"} Nov 03 22:32:20.236709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.236690 4298 scope.go:122] "RemoveContainer" containerID="b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3" Nov 03 22:32:20.236789 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.236757 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:32:20.236868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.236848 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:32:20.236930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.236905 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:32:20.238164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.238140 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:32:20.238250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.238202 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:32:20.238250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.238235 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:32:20.419661 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:20.413504 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:21.254196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:21.253966 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" event={"ID":"d70853f4-2318-4175-9e48-9c8e2b7e0439","Type":"ContainerStarted","Data":"11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1"} Nov 03 22:32:21.259539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:21.259114 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" event={"ID":"221e35d0-71d1-4488-8108-6dcb28e6b621","Type":"ContainerStarted","Data":"f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438"} Nov 03 22:32:21.314385 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:32:21.314347 4298 controller.go:251] "Failed to update lease" err="Put \"https://a3890ba9e34074c9bb3a6df7076aca19-87013095a60c6b5c.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-254.us-west-2.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:21.326987 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:21.326956 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:32:21.327351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:21.327327 4298 patch_prober.go:29] interesting pod/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:32:21.327568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:21.327543 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:32:21.368718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:21.368636 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:32:21.369006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:21.368976 4298 patch_prober.go:29] interesting pod/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:32:21.369108 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:21.369034 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 22:32:22.335143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:22.335095 4298 patch_prober.go:29] interesting pod/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:32:22.339769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:22.339721 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:32:22.369063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:22.369029 4298 patch_prober.go:29] interesting pod/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:32:22.373896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:22.373799 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 22:32:23.345110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:23.345072 4298 patch_prober.go:29] interesting pod/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:32:23.345754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:23.345718 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:32:23.370119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:23.370082 4298 patch_prober.go:29] interesting pod/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:32:23.370417 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:23.370389 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 22:32:24.328923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:24.328882 4298 patch_prober.go:29] interesting pod/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:32:24.329334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:24.329299 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:32:24.372695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:24.372135 4298 patch_prober.go:29] interesting pod/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:32:24.372695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:24.372204 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 22:32:24.859688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:24.857491 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:25.348065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:25.348026 4298 patch_prober.go:29] interesting pod/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:32:25.348360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:25.348331 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:32:25.374140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:25.374104 4298 patch_prober.go:29] interesting pod/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:32:25.374763 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:25.374731 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 22:32:26.336139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:26.336095 4298 patch_prober.go:29] interesting pod/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:32:26.336448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:26.336415 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:32:26.373845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:26.373806 4298 patch_prober.go:29] interesting pod/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:32:26.374159 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:26.374128 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 22:32:27.331261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:27.331233 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:32:27.331539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:27.331519 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:32:27.332136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:27.332116 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:32:27.372174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:27.372139 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:32:27.372507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:27.372484 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:32:27.373163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:27.373141 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:32:29.775869 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:29.775828 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:31.314792 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:32:31.314752 4298 controller.go:251] "Failed to update lease" err="Put \"https://a3890ba9e34074c9bb3a6df7076aca19-87013095a60c6b5c.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-254.us-west-2.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:41.315792 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:32:41.315751 4298 controller.go:251] "Failed to update lease" err="Put \"https://a3890ba9e34074c9bb3a6df7076aca19-87013095a60c6b5c.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-254.us-west-2.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:41.960966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.960924 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.960966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.960924 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8pftz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.960925 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.960980 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.960988 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.960983 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-41bb2b-openshift-default-dockercfg-kqcr4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961034 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3378fd-openshift-default-dockercfg-7fv79\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961041 4298 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961040 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"distinct-2-w5-f81f8f-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961070 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dngqk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961115 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"premium-w2-906673-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961124 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961159 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z5g9w\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961162 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961197 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961200 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961210 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-906673\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961217 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961232 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961249 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961252 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961265 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-41bb2b-dockercfg-k4c7k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961271 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"default-dockercfg-hcclc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961281 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961289 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961314 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961322 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961327 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961352 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961361 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961382 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961392 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g6ff6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961383 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xzmgl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961402 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f7e6db-dockercfg-htx7h\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961434 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c8rtb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961438 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:32:41.961356 4298 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://a3890ba9e34074c9bb3a6df7076aca19-87013095a60c6b5c.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/events\": http2: client connection lost" event=< Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: &Event{ObjectMeta:{e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv.18e4a9b330a8a3c3 openshift-ingress 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-ingress,Name:e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv,UID:221e35d0-71d1-4488-8108-6dcb28e6b621,APIVersion:v1,ResourceVersion:46388,FieldPath:spec.containers{istio-proxy},},Reason:ProbeError,Message:Readiness probe error: Get "http://10.133.0.118:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: body: Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: ,Source:EventSource{Component:kubelet,Host:ip-10-0-139-254.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:32:18.680898499 +0000 UTC m=+1495.661632156,LastTimestamp:2026-11-03 22:32:18.680898499 +0000 UTC m=+1495.661632156,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-254.us-west-2.compute.internal,} Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: > Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961477 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dgskz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961484 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fmrvm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961464 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961501 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961515 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mhkbs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961517 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961532 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961539 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f61714\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961526 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e1c72fb7-dockercfg-x9gxd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961559 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961567 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961557 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jlpf5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:32:41.961553 4298 controller.go:251] "Failed to update lease" err="Put \"https://a3890ba9e34074c9bb3a6df7076aca19-87013095a60c6b5c.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-254.us-west-2.compute.internal?timeout=10s\": http2: client connection lost" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961612 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961618 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961612 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qbrgx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961609 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2d6mp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961639 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961676 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961687 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961692 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961714 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961723 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-28sxb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961729 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961743 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-906673-openshift-default-dockercfg-cwtcb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961767 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961779 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sdlvs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961788 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961781 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f61714-openshift-default-dockercfg-wvcgf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961792 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"default-dockercfg-fr8gq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961814 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961817 4298 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961823 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fvv9p\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961842 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f61714-dockercfg-2k27n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961859 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"distinct-2-w6-6a6b7f-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961874 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961881 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961901 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961940 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"simulator-w5-f81f8f-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961951 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m27qh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961958 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.961981 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962011 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-6a6b7f\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962013 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.961979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962027 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962050 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962082 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962106 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962112 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"default-dockercfg-9jmx4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962149 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f81f8f-openshift-default-dockercfg-mvg86\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962153 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-mt-e1c72fb7-openshift-default-dockercfg-rm7j4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962150 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n7ktg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962161 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"default-dockercfg-hrt5c\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962178 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962185 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-ait-cee4c09d-openshift-default-dockercfg-7z5j6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962191 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962206 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962207 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"simulator-w3-41bb2b-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962226 4298 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962236 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962260 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"default-dockercfg-rpq9b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962303 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-htjcj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962306 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962326 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-28422\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962302 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962313 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962348 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-906673-dockercfg-lcq2m\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962358 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962343 4298 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962369 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cj9c2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962395 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962405 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962438 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962441 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962461 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962472 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962494 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962496 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962512 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962517 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962527 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962520 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962537 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962551 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"simulator-w2-906673-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962568 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962569 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962569 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-6a6b7f-dockercfg-dcsr2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962589 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962603 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fnhf4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962614 4298 controller.go:171] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962621 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962630 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962673 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962605 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"simulator-w6-6a6b7f-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962695 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962624 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962702 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962673 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962719 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-ng8jg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962724 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-pcob6ei5ut7v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962732 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962605 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j54w4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962719 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962745 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962621 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962759 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962638 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bkplg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962773 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fcb7n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962778 4298 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962723 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962770 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962637 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962799 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962695 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"distinct-w5-f81f8f-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962810 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962703 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gsv4s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962820 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e1c72fb7\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962828 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962784 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962837 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962840 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-41bb2b\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962846 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962760 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962848 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962863 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f7e6db-openshift-default-dockercfg-dfr7z\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962880 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962890 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962895 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5mdhj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962873 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962906 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-wxs6m\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962882 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962917 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hfgqt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962930 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"default-dockercfg-kdmdn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962934 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f7e6db\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962762 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962961 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.964529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962906 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-migrate-59a07a45-gw-openshift-default-dockercfg-ckmcx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962968 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbwbg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962896 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ckrtj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962936 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ng2g2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962797 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962920 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6a6b7f-openshift-default-dockercfg-fqnpk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962990 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"premium-w5-f81f8f-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962960 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.963003 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962962 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.963013 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962976 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962988 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962964 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.963004 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-d2krk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.963006 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962934 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962864 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.963017 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kxpq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.963019 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"distinct-2-w1-3378fd-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:41.967381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:41.962892 4298 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"premium-w0-f61714-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 22:32:42.614539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.614504 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:42.784999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.784959 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:42.829024 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.828991 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:32:42.839384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.839358 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:32:42.879597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.879509 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:32:42.879597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.879524 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hfgqt\"" Nov 03 22:32:42.879597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.879549 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e1c72fb7\"" Nov 03 22:32:42.889815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.889787 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 22:32:42.889976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.889836 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:32:42.889976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.889860 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 22:32:42.889976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.889945 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"premium-w2-906673-kserve-self-signed-certs\"" Nov 03 22:32:42.929856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.929828 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f7e6db-openshift-default-dockercfg-dfr7z\"" Nov 03 22:32:42.930054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.929876 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 22:32:42.930054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.929994 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"default-dockercfg-hcclc\"" Nov 03 22:32:42.930054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.930051 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sdlvs\"" Nov 03 22:32:42.930182 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.930070 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f7e6db-dockercfg-htx7h\"" Nov 03 22:32:42.940456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.940427 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 22:32:42.940456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.940427 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z5g9w\"" Nov 03 22:32:42.940633 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:42.940451 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.041972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.041930 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 22:32:43.041972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.041940 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.041972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.041973 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:32:43.041972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.041974 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 22:32:43.042322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.041992 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 22:32:43.042322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042011 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:32:43.042322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042118 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.042322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042125 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5mdhj\"" Nov 03 22:32:43.042322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042201 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fcb7n\"" Nov 03 22:32:43.042322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042218 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:32:43.042322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042262 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.042322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042288 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:32:43.042322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042305 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 22:32:43.042322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042311 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-906673-openshift-default-dockercfg-cwtcb\"" Nov 03 22:32:43.042322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042306 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.042778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042345 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.042778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042356 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.042778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042373 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ng2g2\"" Nov 03 22:32:43.042778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042397 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f81f8f-openshift-default-dockercfg-mvg86\"" Nov 03 22:32:43.042778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042412 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fmrvm\"" Nov 03 22:32:43.042778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.042509 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 22:32:43.081045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.081009 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gsv4s\"" Nov 03 22:32:43.081339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.081321 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-d2krk\"" Nov 03 22:32:43.081436 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.081409 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.081523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.081433 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.081523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.081444 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 22:32:43.081523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.081448 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.081712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.081557 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"distinct-2-w5-f81f8f-kserve-self-signed-certs\"" Nov 03 22:32:43.081712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.081591 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-41bb2b\"" Nov 03 22:32:43.081712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.081699 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.086403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.086390 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.088555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.088531 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:32:43.093052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.093030 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"default-dockercfg-kdmdn\"" Nov 03 22:32:43.093144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.093065 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 22:32:43.094481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.094464 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:32:43.096984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.096968 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.101908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.101892 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.103121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.103107 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:32:43.103214 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.103144 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.105244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.105216 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-28422\"" Nov 03 22:32:43.115669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.115620 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f61714-dockercfg-2k27n\"" Nov 03 22:32:43.115669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.115635 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:32:43.117692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.117676 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fvv9p\"" Nov 03 22:32:43.122114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.122095 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 22:32:43.122286 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.122269 4298 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:32:43.129002 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.128984 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-ng8jg\"" Nov 03 22:32:43.132236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.132184 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 22:32:43.142032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.142004 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"distinct-2-w6-6a6b7f-kserve-self-signed-certs\"" Nov 03 22:32:43.142851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.142836 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:32:43.147841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.147820 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g6ff6\"" Nov 03 22:32:43.151413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.151378 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 22:32:43.153098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.153074 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"simulator-w3-41bb2b-kserve-self-signed-certs\"" Nov 03 22:32:43.155290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.155265 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-906673\"" Nov 03 22:32:43.162099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.162077 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.165324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.165305 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-41bb2b-dockercfg-k4c7k\"" Nov 03 22:32:43.172661 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.170972 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 22:32:43.179662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.178481 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qbrgx\"" Nov 03 22:32:43.181729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.181704 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.184509 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.184481 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 22:32:43.186887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.186868 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.189250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.189228 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:32:43.189393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.189242 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 22:32:43.190371 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.190353 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbwbg\"" Nov 03 22:32:43.192872 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.192853 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.193060 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.193045 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f61714\"" Nov 03 22:32:43.196264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.196250 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.197342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.197320 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 22:32:43.202787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.202769 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c8rtb\"" Nov 03 22:32:43.206054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.206032 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:32:43.223443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.223312 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 22:32:43.223443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.223354 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 22:32:43.223741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.223487 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:32:43.225000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.224576 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.227735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.227716 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"simulator-w6-6a6b7f-kserve-self-signed-certs\"" Nov 03 22:32:43.238667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.238616 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-6a6b7f-dockercfg-dcsr2\"" Nov 03 22:32:43.250279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.250246 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.254053 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.254034 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:32:43.258071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.258002 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 22:32:43.258895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.258034 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f7e6db\"" Nov 03 22:32:43.266081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.266059 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8pftz\"" Nov 03 22:32:43.274115 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.274093 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:32:43.274460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.274442 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cj9c2\"" Nov 03 22:32:43.274517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.274484 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.278751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.278733 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"default-dockercfg-fr8gq\"" Nov 03 22:32:43.282186 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.282167 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"simulator-w5-f81f8f-kserve-self-signed-certs\"" Nov 03 22:32:43.286178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.286161 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:32:43.286513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.286490 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 22:32:43.302266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.302227 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e1c72fb7-dockercfg-x9gxd\"" Nov 03 22:32:43.302975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.302957 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:32:43.312751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.312729 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:32:43.317362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.317341 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dgskz\"" Nov 03 22:32:43.317446 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.317408 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"premium-w0-f61714-kserve-self-signed-certs\"" Nov 03 22:32:43.319494 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.319478 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.324606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.324590 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m27qh\"" Nov 03 22:32:43.329778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.329757 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"distinct-2-w1-3378fd-kserve-self-signed-certs\"" Nov 03 22:32:43.341485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.339192 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.346565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.346541 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-wxs6m\"" Nov 03 22:32:43.352865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.352841 4298 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:32:43.352865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.352847 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-41bb2b-openshift-default-dockercfg-kqcr4\"" Nov 03 22:32:43.353424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.353409 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.353523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.353445 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.366347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.366326 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jlpf5\"" Nov 03 22:32:43.366456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.366389 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:32:43.369734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.369714 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.371866 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.371849 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n7ktg\"" Nov 03 22:32:43.372928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.372914 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.376802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.376786 4298 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:32:43.380229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.380214 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-pcob6ei5ut7v\"" Nov 03 22:32:43.384637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.384591 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"premium-w5-f81f8f-kserve-self-signed-certs\"" Nov 03 22:32:43.387680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.387652 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 22:32:43.390498 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.390474 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 22:32:43.396739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.396719 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ckrtj\"" Nov 03 22:32:43.401936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.401919 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 22:32:43.403138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.403120 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-htjcj\"" Nov 03 22:32:43.403240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.403226 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mhkbs\"" Nov 03 22:32:43.407906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.407883 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 22:32:43.411980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.411962 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"default-dockercfg-hrt5c\"" Nov 03 22:32:43.412237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.412223 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 22:32:43.414087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.414073 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:32:43.422453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.422432 4298 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:32:43.423069 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.423055 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 22:32:43.458346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.458309 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 22:32:43.467079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.466901 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6a6b7f-openshift-default-dockercfg-fqnpk\"" Nov 03 22:32:43.467079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.466946 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:32:43.467079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.466948 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 22:32:43.467079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.466974 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cee4c09d-openshift-default-dockercfg-7z5j6\"" Nov 03 22:32:43.467079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.467024 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-6a6b7f\"" Nov 03 22:32:43.467079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.467083 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3378fd-openshift-default-dockercfg-7fv79\"" Nov 03 22:32:43.467514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.467097 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.467514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.467164 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"default-dockercfg-rpq9b\"" Nov 03 22:32:43.467514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.467374 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.467514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.467399 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:32:43.467514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.467440 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"default-dockercfg-9jmx4\"" Nov 03 22:32:43.467710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.467526 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bkplg\"" Nov 03 22:32:43.469857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.469838 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 22:32:43.473686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.473671 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.477222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.477207 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j54w4\"" Nov 03 22:32:43.478292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.478278 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 22:32:43.478383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.478370 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dngqk\"" Nov 03 22:32:43.482972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.482958 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kxpq\"" Nov 03 22:32:43.484327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.484313 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"distinct-w5-f81f8f-kserve-self-signed-certs\"" Nov 03 22:32:43.493793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.493775 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-906673-dockercfg-lcq2m\"" Nov 03 22:32:43.498240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.498219 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.505233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.505214 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:32:43.506156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.506139 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fnhf4\"" Nov 03 22:32:43.506486 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.506470 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-59a07a45-gw-openshift-default-dockercfg-ckmcx\"" Nov 03 22:32:43.508920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.508906 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-28sxb\"" Nov 03 22:32:43.527141 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.527117 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xzmgl\"" Nov 03 22:32:43.537949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.537920 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e1c72fb7-openshift-default-dockercfg-rm7j4\"" Nov 03 22:32:43.546186 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.546163 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 22:32:43.552411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.552384 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 22:32:43.555627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.555606 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.557129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.557106 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 22:32:43.558604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.558583 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 22:32:43.559819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.559789 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:43.562244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.562223 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 22:32:43.562603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.562588 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2d6mp\"" Nov 03 22:32:43.562730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.562634 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f61714-openshift-default-dockercfg-wvcgf\"" Nov 03 22:32:43.563402 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.563386 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"openshift-service-ca.crt\"" Nov 03 22:32:43.563903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.563885 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 22:32:43.570836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.570808 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 22:32:43.574995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.574974 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:32:43.596941 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.596902 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 22:32:43.597155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.597134 4298 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:32:43.603068 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.603035 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.603068 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.603035 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"simulator-w2-906673-kserve-self-signed-certs\"" Nov 03 22:32:43.603908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.603832 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.604222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.604200 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs","odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg","odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b","odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9"] Nov 03 22:32:43.606155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.606126 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 22:32:43.606300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.606168 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 22:32:43.606775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.606757 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.607620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.607602 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.607620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.607602 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.610846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.610820 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.613355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.613333 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f81f8f\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.613534 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.613429 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:32:43.632681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.617777 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:43.632681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.627910 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:43.635038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.634726 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:43.653823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.650948 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:43.657457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.655886 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:43.657457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.655952 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3937145-g8t6b\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:43.657457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.656042 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:43.657457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.656233 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc5gn\" (UniqueName: \"kubernetes.io/projected/f64db09e-4b9e-4530-a2fe-741a08d47c74-kube-api-access-gc5gn\") pod \"maas-api-revoke-keys-e2e-derive-e3937145-g8t6b\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:43.657457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.656334 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqwwx\" (UniqueName: \"kubernetes.io/projected/d0f7c114-f196-4de1-88f3-4878834ee308-kube-api-access-pqwwx\") pod \"maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:43.657457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.656421 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3937145-g8t6b\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:43.675999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.674825 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:32:43.675999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.675569 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:32:43.755812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.755716 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:43.757751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.757688 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqwwx\" (UniqueName: \"kubernetes.io/projected/d0f7c114-f196-4de1-88f3-4878834ee308-kube-api-access-pqwwx\") pod \"maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:43.757751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.757748 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv5zq\" (UniqueName: \"kubernetes.io/projected/c87b60aa-361b-472b-adcc-d2ca98a4c11a-kube-api-access-tv5zq\") pod \"maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:43.757979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.757811 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3937145-g8t6b\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:43.757979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.757901 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:43.757979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.757932 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3937145-g8t6b\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:43.758140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.757996 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:43.758140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.758040 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:43.758140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.758099 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:43.758351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.758152 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:43.758351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.758229 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjjb\" (UniqueName: \"kubernetes.io/projected/6e212a5a-491d-4410-8189-20b3c53129bd-kube-api-access-5zjjb\") pod \"maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:43.758351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.758258 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc5gn\" (UniqueName: \"kubernetes.io/projected/f64db09e-4b9e-4530-a2fe-741a08d47c74-kube-api-access-gc5gn\") pod \"maas-api-revoke-keys-e2e-derive-e3937145-g8t6b\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:43.758351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.758287 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:43.759982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.759956 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:43.775331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.760825 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3937145-g8t6b\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:43.782305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.782216 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:43.782949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.782867 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3937145-g8t6b\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:43.815814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.815779 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqwwx\" (UniqueName: \"kubernetes.io/projected/d0f7c114-f196-4de1-88f3-4878834ee308-kube-api-access-pqwwx\") pod \"maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:43.832552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.832524 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc5gn\" (UniqueName: \"kubernetes.io/projected/f64db09e-4b9e-4530-a2fe-741a08d47c74-kube-api-access-gc5gn\") pod \"maas-api-revoke-keys-e2e-derive-e3937145-g8t6b\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:43.862217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.862172 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjjb\" (UniqueName: \"kubernetes.io/projected/6e212a5a-491d-4410-8189-20b3c53129bd-kube-api-access-5zjjb\") pod \"maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:43.862553 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.862242 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:43.862796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.862767 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv5zq\" (UniqueName: \"kubernetes.io/projected/c87b60aa-361b-472b-adcc-d2ca98a4c11a-kube-api-access-tv5zq\") pod \"maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:43.865327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.863044 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:43.865327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.863122 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:43.865327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.863183 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:43.884616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.866288 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:43.884616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.868954 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:43.884616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.870569 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:43.892415 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.892325 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:43.919204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.919171 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjjb\" (UniqueName: \"kubernetes.io/projected/6e212a5a-491d-4410-8189-20b3c53129bd-kube-api-access-5zjjb\") pod \"maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:43.923510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:43.923479 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv5zq\" (UniqueName: \"kubernetes.io/projected/c87b60aa-361b-472b-adcc-d2ca98a4c11a-kube-api-access-tv5zq\") pod \"maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:44.013080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:44.013040 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:44.033896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:44.033861 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:44.066021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:44.065987 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:44.114687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:44.114624 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:44.393858 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:32:44.393825 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f7c114_f196_4de1_88f3_4878834ee308.slice/crio-733f3199857fb5cd48059b3be959b9b28523528641ee2cc631d0f585e49c8b1f WatchSource:0}: Error finding container 733f3199857fb5cd48059b3be959b9b28523528641ee2cc631d0f585e49c8b1f: Status 404 returned error can't find the container with id 733f3199857fb5cd48059b3be959b9b28523528641ee2cc631d0f585e49c8b1f Nov 03 22:32:44.399420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:44.399397 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs"] Nov 03 22:32:44.441092 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:32:44.441061 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64db09e_4b9e_4530_a2fe_741a08d47c74.slice/crio-ce99a01aa352315e3af1b686dc8daa22616db390dce9f0a4c4313f6c672f10a8 WatchSource:0}: Error finding container ce99a01aa352315e3af1b686dc8daa22616db390dce9f0a4c4313f6c672f10a8: Status 404 returned error can't find the container with id ce99a01aa352315e3af1b686dc8daa22616db390dce9f0a4c4313f6c672f10a8 Nov 03 22:32:44.447315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:44.447288 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b"] Nov 03 22:32:44.452825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:44.452796 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" event={"ID":"f64db09e-4b9e-4530-a2fe-741a08d47c74","Type":"ContainerStarted","Data":"ce99a01aa352315e3af1b686dc8daa22616db390dce9f0a4c4313f6c672f10a8"} Nov 03 22:32:44.454160 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:44.454137 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" event={"ID":"d0f7c114-f196-4de1-88f3-4878834ee308","Type":"ContainerStarted","Data":"733f3199857fb5cd48059b3be959b9b28523528641ee2cc631d0f585e49c8b1f"} Nov 03 22:32:44.487759 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:32:44.487724 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87b60aa_361b_472b_adcc_d2ca98a4c11a.slice/crio-edff599c36ceba46ce86e076557d543219badb9cf16e7a4479e28be8503bc694 WatchSource:0}: Error finding container edff599c36ceba46ce86e076557d543219badb9cf16e7a4479e28be8503bc694: Status 404 returned error can't find the container with id edff599c36ceba46ce86e076557d543219badb9cf16e7a4479e28be8503bc694 Nov 03 22:32:44.493595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:44.493568 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9"] Nov 03 22:32:44.525483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:44.525455 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg"] Nov 03 22:32:44.527205 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:32:44.527175 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e212a5a_491d_4410_8189_20b3c53129bd.slice/crio-948b89ecaba1a2e19e24935505eac898a3c753896e33c2b00ca2b047eef9221d WatchSource:0}: Error finding container 948b89ecaba1a2e19e24935505eac898a3c753896e33c2b00ca2b047eef9221d: Status 404 returned error can't find the container with id 948b89ecaba1a2e19e24935505eac898a3c753896e33c2b00ca2b047eef9221d Nov 03 22:32:45.465213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:45.465182 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" event={"ID":"c87b60aa-361b-472b-adcc-d2ca98a4c11a","Type":"ContainerStarted","Data":"f42de6f57666696665987127075b9441f025c8fc8d4424f47866a0e7d192f6f0"} Nov 03 22:32:45.465579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:45.465242 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" event={"ID":"c87b60aa-361b-472b-adcc-d2ca98a4c11a","Type":"ContainerStarted","Data":"edff599c36ceba46ce86e076557d543219badb9cf16e7a4479e28be8503bc694"} Nov 03 22:32:45.466956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:45.466931 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" event={"ID":"6e212a5a-491d-4410-8189-20b3c53129bd","Type":"ContainerStarted","Data":"1338a7530bd79a258ce3b0910fbb8453c7d2a36fffb5e80be5050b5dc44f7519"} Nov 03 22:32:45.467048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:45.466996 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" event={"ID":"6e212a5a-491d-4410-8189-20b3c53129bd","Type":"ContainerStarted","Data":"948b89ecaba1a2e19e24935505eac898a3c753896e33c2b00ca2b047eef9221d"} Nov 03 22:32:45.468477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:45.468457 4298 generic.go:356] "Generic (PLEG): container finished" podID="f64db09e-4b9e-4530-a2fe-741a08d47c74" containerID="889a5753db6a4fb59e22fb67a520620f1f83680cf32bd589912604072e9eb0fd" exitCode=0 Nov 03 22:32:45.468559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:45.468516 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" event={"ID":"f64db09e-4b9e-4530-a2fe-741a08d47c74","Type":"ContainerDied","Data":"889a5753db6a4fb59e22fb67a520620f1f83680cf32bd589912604072e9eb0fd"} Nov 03 22:32:45.470034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:45.470010 4298 generic.go:356] "Generic (PLEG): container finished" podID="d0f7c114-f196-4de1-88f3-4878834ee308" containerID="cb055a8ada08f007de06c464106c65fc65811a65d699bdf0dcd9c21cae790af7" exitCode=0 Nov 03 22:32:45.470116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:45.470077 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" event={"ID":"d0f7c114-f196-4de1-88f3-4878834ee308","Type":"ContainerDied","Data":"cb055a8ada08f007de06c464106c65fc65811a65d699bdf0dcd9c21cae790af7"} Nov 03 22:32:45.581944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:45.581885 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" podStartSLOduration=28.581871994 podStartE2EDuration="28.581871994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:45.579597804 +0000 UTC m=+1522.560331456" watchObservedRunningTime="2026-11-03 22:32:45.581871994 +0000 UTC m=+1522.562605647" Nov 03 22:32:45.917665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:45.917409 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" podStartSLOduration=18.917385328 podStartE2EDuration="18.917385328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:45.809768014 +0000 UTC m=+1522.790501668" watchObservedRunningTime="2026-11-03 22:32:45.917385328 +0000 UTC m=+1522.898118987" Nov 03 22:32:46.479504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.479470 4298 generic.go:356] "Generic (PLEG): container finished" podID="6e212a5a-491d-4410-8189-20b3c53129bd" containerID="1338a7530bd79a258ce3b0910fbb8453c7d2a36fffb5e80be5050b5dc44f7519" exitCode=0 Nov 03 22:32:46.480015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.479561 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" event={"ID":"6e212a5a-491d-4410-8189-20b3c53129bd","Type":"ContainerDied","Data":"1338a7530bd79a258ce3b0910fbb8453c7d2a36fffb5e80be5050b5dc44f7519"} Nov 03 22:32:46.481515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.481492 4298 generic.go:356] "Generic (PLEG): container finished" podID="c87b60aa-361b-472b-adcc-d2ca98a4c11a" containerID="f42de6f57666696665987127075b9441f025c8fc8d4424f47866a0e7d192f6f0" exitCode=0 Nov 03 22:32:46.481674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.481580 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" event={"ID":"c87b60aa-361b-472b-adcc-d2ca98a4c11a","Type":"ContainerDied","Data":"f42de6f57666696665987127075b9441f025c8fc8d4424f47866a0e7d192f6f0"} Nov 03 22:32:46.684180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.684126 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:46.733496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.733415 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:46.826198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.826154 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-xks-ca\") pod \"f64db09e-4b9e-4530-a2fe-741a08d47c74\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " Nov 03 22:32:46.826387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.826237 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-xks-ca\") pod \"d0f7c114-f196-4de1-88f3-4878834ee308\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " Nov 03 22:32:46.826387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.826318 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f7c114-f196-4de1-88f3-4878834ee308-kube-api-access-pqwwx\" (UniqueName: \"kubernetes.io/projected/d0f7c114-f196-4de1-88f3-4878834ee308-kube-api-access-pqwwx\") pod \"d0f7c114-f196-4de1-88f3-4878834ee308\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " Nov 03 22:32:46.826510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.826411 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f64db09e-4b9e-4530-a2fe-741a08d47c74-kube-api-access-gc5gn\" (UniqueName: \"kubernetes.io/projected/f64db09e-4b9e-4530-a2fe-741a08d47c74-kube-api-access-gc5gn\") pod \"f64db09e-4b9e-4530-a2fe-741a08d47c74\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " Nov 03 22:32:46.826510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.826465 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-openshift-ca\") pod \"d0f7c114-f196-4de1-88f3-4878834ee308\" (UID: \"d0f7c114-f196-4de1-88f3-4878834ee308\") " Nov 03 22:32:46.826620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.826537 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-openshift-ca\") pod \"f64db09e-4b9e-4530-a2fe-741a08d47c74\" (UID: \"f64db09e-4b9e-4530-a2fe-741a08d47c74\") " Nov 03 22:32:46.827981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.827948 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-openshift-ca" pod "d0f7c114-f196-4de1-88f3-4878834ee308" (UID: "d0f7c114-f196-4de1-88f3-4878834ee308"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:32:46.828130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.827980 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-openshift-ca" pod "f64db09e-4b9e-4530-a2fe-741a08d47c74" (UID: "f64db09e-4b9e-4530-a2fe-741a08d47c74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:32:46.831796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.831753 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-xks-ca" pod "d0f7c114-f196-4de1-88f3-4878834ee308" (UID: "d0f7c114-f196-4de1-88f3-4878834ee308"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:32:46.832290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.832262 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-xks-ca" pod "f64db09e-4b9e-4530-a2fe-741a08d47c74" (UID: "f64db09e-4b9e-4530-a2fe-741a08d47c74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:32:46.833131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.833107 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f7c114-f196-4de1-88f3-4878834ee308-kube-api-access-pqwwx" pod "d0f7c114-f196-4de1-88f3-4878834ee308" (UID: "d0f7c114-f196-4de1-88f3-4878834ee308"). InnerVolumeSpecName "kube-api-access-pqwwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:46.840777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.840743 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64db09e-4b9e-4530-a2fe-741a08d47c74-kube-api-access-gc5gn" pod "f64db09e-4b9e-4530-a2fe-741a08d47c74" (UID: "f64db09e-4b9e-4530-a2fe-741a08d47c74"). InnerVolumeSpecName "kube-api-access-gc5gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:46.929145 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.929107 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:46.929145 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.929138 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:46.929356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.929151 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqwwx\" (UniqueName: \"kubernetes.io/projected/d0f7c114-f196-4de1-88f3-4878834ee308-kube-api-access-pqwwx\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:46.929356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.929166 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc5gn\" (UniqueName: \"kubernetes.io/projected/f64db09e-4b9e-4530-a2fe-741a08d47c74-kube-api-access-gc5gn\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:46.929356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.929175 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f7c114-f196-4de1-88f3-4878834ee308-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:46.929356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:46.929185 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64db09e-4b9e-4530-a2fe-741a08d47c74-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:47.492613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.492582 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" Nov 03 22:32:47.493089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.492582 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b" event={"ID":"f64db09e-4b9e-4530-a2fe-741a08d47c74","Type":"ContainerDied","Data":"ce99a01aa352315e3af1b686dc8daa22616db390dce9f0a4c4313f6c672f10a8"} Nov 03 22:32:47.493089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.492727 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce99a01aa352315e3af1b686dc8daa22616db390dce9f0a4c4313f6c672f10a8" Nov 03 22:32:47.494162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.494140 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" Nov 03 22:32:47.494291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.494161 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs" event={"ID":"d0f7c114-f196-4de1-88f3-4878834ee308","Type":"ContainerDied","Data":"733f3199857fb5cd48059b3be959b9b28523528641ee2cc631d0f585e49c8b1f"} Nov 03 22:32:47.494291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.494205 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="733f3199857fb5cd48059b3be959b9b28523528641ee2cc631d0f585e49c8b1f" Nov 03 22:32:47.564231 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:32:47.561800 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:32:47.936532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.936503 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:47.941907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.941880 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:47.959471 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.959438 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-xks-ca\") pod \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " Nov 03 22:32:47.959664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.959571 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e212a5a-491d-4410-8189-20b3c53129bd-kube-api-access-5zjjb\" (UniqueName: \"kubernetes.io/projected/6e212a5a-491d-4410-8189-20b3c53129bd-kube-api-access-5zjjb\") pod \"6e212a5a-491d-4410-8189-20b3c53129bd\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " Nov 03 22:32:47.959664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.959596 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-xks-ca\") pod \"6e212a5a-491d-4410-8189-20b3c53129bd\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " Nov 03 22:32:47.959804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.959713 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c87b60aa-361b-472b-adcc-d2ca98a4c11a-kube-api-access-tv5zq\" (UniqueName: \"kubernetes.io/projected/c87b60aa-361b-472b-adcc-d2ca98a4c11a-kube-api-access-tv5zq\") pod \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " Nov 03 22:32:47.959804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.959758 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-openshift-ca\") pod \"6e212a5a-491d-4410-8189-20b3c53129bd\" (UID: \"6e212a5a-491d-4410-8189-20b3c53129bd\") " Nov 03 22:32:47.959933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.959891 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-openshift-ca\") pod \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\" (UID: \"c87b60aa-361b-472b-adcc-d2ca98a4c11a\") " Nov 03 22:32:47.960390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.960348 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-openshift-ca" pod "6e212a5a-491d-4410-8189-20b3c53129bd" (UID: "6e212a5a-491d-4410-8189-20b3c53129bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:32:47.960544 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.960428 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-openshift-ca" pod "c87b60aa-361b-472b-adcc-d2ca98a4c11a" (UID: "c87b60aa-361b-472b-adcc-d2ca98a4c11a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:32:47.960962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.960936 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:47.961054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.960970 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:47.962335 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.962297 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e212a5a-491d-4410-8189-20b3c53129bd-kube-api-access-5zjjb" pod "6e212a5a-491d-4410-8189-20b3c53129bd" (UID: "6e212a5a-491d-4410-8189-20b3c53129bd"). InnerVolumeSpecName "kube-api-access-5zjjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:47.962434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.962337 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87b60aa-361b-472b-adcc-d2ca98a4c11a-kube-api-access-tv5zq" pod "c87b60aa-361b-472b-adcc-d2ca98a4c11a" (UID: "c87b60aa-361b-472b-adcc-d2ca98a4c11a"). InnerVolumeSpecName "kube-api-access-tv5zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:47.962434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.962371 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-xks-ca" pod "c87b60aa-361b-472b-adcc-d2ca98a4c11a" (UID: "c87b60aa-361b-472b-adcc-d2ca98a4c11a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:32:47.962527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:47.962452 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-xks-ca" pod "6e212a5a-491d-4410-8189-20b3c53129bd" (UID: "6e212a5a-491d-4410-8189-20b3c53129bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:32:48.062721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:48.062583 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv5zq\" (UniqueName: \"kubernetes.io/projected/c87b60aa-361b-472b-adcc-d2ca98a4c11a-kube-api-access-tv5zq\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:48.062721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:48.062627 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87b60aa-361b-472b-adcc-d2ca98a4c11a-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:48.062721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:48.062660 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zjjb\" (UniqueName: \"kubernetes.io/projected/6e212a5a-491d-4410-8189-20b3c53129bd-kube-api-access-5zjjb\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:48.062721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:48.062676 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e212a5a-491d-4410-8189-20b3c53129bd-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:32:48.108893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:48.108862 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:48.503734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:48.503704 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" Nov 03 22:32:48.504157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:48.503755 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg" event={"ID":"6e212a5a-491d-4410-8189-20b3c53129bd","Type":"ContainerDied","Data":"948b89ecaba1a2e19e24935505eac898a3c753896e33c2b00ca2b047eef9221d"} Nov 03 22:32:48.504157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:48.503809 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="948b89ecaba1a2e19e24935505eac898a3c753896e33c2b00ca2b047eef9221d" Nov 03 22:32:48.505296 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:48.505277 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" Nov 03 22:32:48.505406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:48.505319 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9" event={"ID":"c87b60aa-361b-472b-adcc-d2ca98a4c11a","Type":"ContainerDied","Data":"edff599c36ceba46ce86e076557d543219badb9cf16e7a4479e28be8503bc694"} Nov 03 22:32:48.505406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:48.505355 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edff599c36ceba46ce86e076557d543219badb9cf16e7a4479e28be8503bc694" Nov 03 22:32:52.775328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:52.775294 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:55.797238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:55.797198 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:55.965581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:55.965543 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:58.007248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:32:58.007207 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:00.991414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:00.991371 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:01.173394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:01.173349 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:02.215227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:02.215186 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:02.438367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:02.438335 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:02.684527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:02.684432 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:02.874574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:02.874525 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:03.374707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:03.372107 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:03.593420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:03.593363 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:03.814929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:03.814875 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:04.030232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:04.030195 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:04.839048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:04.839013 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:06.560712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:06.560672 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:06.831020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:06.830933 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:07.015086 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:07.015053 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:07.138401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:07.138304 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b"] Nov 03 22:33:07.186599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:07.186567 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3937145-g8t6b"] Nov 03 22:33:07.274394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:07.274361 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:08.276093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:08.276056 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:08.458327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:08.458297 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:08.656828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:08.656727 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:08.842292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:08.842265 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:08.974946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:08.974909 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9"] Nov 03 22:33:09.030578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:09.030546 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cee4c09d-fwgm9"] Nov 03 22:33:09.041141 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:09.041111 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m"] Nov 03 22:33:09.044769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:09.044499 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" containerID="cri-o://3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9" gracePeriod=30 Nov 03 22:33:09.089486 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:09.089457 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:09.190663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:09.189292 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c87b60aa-361b-472b-adcc-d2ca98a4c11a" path="/var/lib/kubelet/pods/c87b60aa-361b-472b-adcc-d2ca98a4c11a/volumes" Nov 03 22:33:09.190663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:09.190056 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f64db09e-4b9e-4530-a2fe-741a08d47c74" path="/var/lib/kubelet/pods/f64db09e-4b9e-4530-a2fe-741a08d47c74/volumes" Nov 03 22:33:09.284465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:09.284384 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:10.027508 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:10.027479 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:10.209924 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:10.209869 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:10.605386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:10.605357 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:10.789361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:10.789332 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:11.082664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.068511 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws"] Nov 03 22:33:11.125844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.125795 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64455612-fx2ws"] Nov 03 22:33:11.141205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.141166 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777d8e0a-571f-4941-b6e6-45aa67036eff" path="/var/lib/kubelet/pods/777d8e0a-571f-4941-b6e6-45aa67036eff/volumes" Nov 03 22:33:11.364453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.364373 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj"] Nov 03 22:33:11.365074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365059 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f64db09e-4b9e-4530-a2fe-741a08d47c74" containerName="revoke-keys" Nov 03 22:33:11.365116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365075 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f64db09e-4b9e-4530-a2fe-741a08d47c74" containerName="revoke-keys" Nov 03 22:33:11.365116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365105 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d0f7c114-f196-4de1-88f3-4878834ee308" containerName="revoke-keys" Nov 03 22:33:11.365116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365110 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f7c114-f196-4de1-88f3-4878834ee308" containerName="revoke-keys" Nov 03 22:33:11.365217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365117 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c87b60aa-361b-472b-adcc-d2ca98a4c11a" containerName="revoke-keys" Nov 03 22:33:11.365217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365123 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87b60aa-361b-472b-adcc-d2ca98a4c11a" containerName="revoke-keys" Nov 03 22:33:11.365217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365134 4298 cpu_manager.go:414] "containerMap: removing container" podUID="6e212a5a-491d-4410-8189-20b3c53129bd" containerName="revoke-keys" Nov 03 22:33:11.365217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365139 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e212a5a-491d-4410-8189-20b3c53129bd" containerName="revoke-keys" Nov 03 22:33:11.365357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365225 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f7c114-f196-4de1-88f3-4878834ee308" containerName="revoke-keys" Nov 03 22:33:11.365357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365235 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87b60aa-361b-472b-adcc-d2ca98a4c11a" containerName="revoke-keys" Nov 03 22:33:11.365357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365243 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e212a5a-491d-4410-8189-20b3c53129bd" containerName="revoke-keys" Nov 03 22:33:11.365357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.365253 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f64db09e-4b9e-4530-a2fe-741a08d47c74" containerName="revoke-keys" Nov 03 22:33:11.370482 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.370465 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:11.422878 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.422847 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj"] Nov 03 22:33:11.464269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.464240 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:33:11.464534 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.464511 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:33:11.494419 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.494380 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:11.494609 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.494540 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr79z\" (UniqueName: \"kubernetes.io/projected/a3f73568-8372-4f53-819c-81b7891c60ee-kube-api-access-tr79z\") pod \"maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:11.494708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.494605 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:11.592459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.592425 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:11.596146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.596121 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:11.596257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.596241 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr79z\" (UniqueName: \"kubernetes.io/projected/a3f73568-8372-4f53-819c-81b7891c60ee-kube-api-access-tr79z\") pod \"maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:11.596331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.596290 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:11.597176 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.597148 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:11.598472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.598451 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:11.653784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.653706 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr79z\" (UniqueName: \"kubernetes.io/projected/a3f73568-8372-4f53-819c-81b7891c60ee-kube-api-access-tr79z\") pod \"maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:11.685543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.685506 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:11.787665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:11.782869 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:12.189431 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:33:12.189397 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f73568_8372_4f53_819c_81b7891c60ee.slice/crio-4d868d438dff211a13cb673e4d055ed782e59e41e1353823149c8dd3654c0e5a WatchSource:0}: Error finding container 4d868d438dff211a13cb673e4d055ed782e59e41e1353823149c8dd3654c0e5a: Status 404 returned error can't find the container with id 4d868d438dff211a13cb673e4d055ed782e59e41e1353823149c8dd3654c0e5a Nov 03 22:33:12.195020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:12.194833 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj"] Nov 03 22:33:12.942899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:12.942815 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" event={"ID":"a3f73568-8372-4f53-819c-81b7891c60ee","Type":"ContainerStarted","Data":"2f17f61fa2e565f50dede7dd20fc1c2666bf673f5a792b7e759517937d4cbc3a"} Nov 03 22:33:12.942899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:12.942870 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" event={"ID":"a3f73568-8372-4f53-819c-81b7891c60ee","Type":"ContainerStarted","Data":"4d868d438dff211a13cb673e4d055ed782e59e41e1353823149c8dd3654c0e5a"} Nov 03 22:33:12.984587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:12.984556 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:13.063611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:13.063555 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" podStartSLOduration=2.063539638 podStartE2EDuration="2.063539638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:13.057024324 +0000 UTC m=+1550.037758010" watchObservedRunningTime="2026-11-03 22:33:13.063539638 +0000 UTC m=+1550.044273290" Nov 03 22:33:13.172111 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:13.172058 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:13.357932 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:13.357889 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:13.457739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:13.457689 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs"] Nov 03 22:33:13.465370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:13.465335 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7831d-vdpfs"] Nov 03 22:33:13.635203 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:13.635102 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:13.952372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:13.952334 4298 generic.go:356] "Generic (PLEG): container finished" podID="a3f73568-8372-4f53-819c-81b7891c60ee" containerID="2f17f61fa2e565f50dede7dd20fc1c2666bf673f5a792b7e759517937d4cbc3a" exitCode=0 Nov 03 22:33:13.952776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:13.952420 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" event={"ID":"a3f73568-8372-4f53-819c-81b7891c60ee","Type":"ContainerDied","Data":"2f17f61fa2e565f50dede7dd20fc1c2666bf673f5a792b7e759517937d4cbc3a"} Nov 03 22:33:14.035399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.035366 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:14.324808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.324778 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:33:14.362803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.362694 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:14.452956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.452925 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-certs\") pod \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " Nov 03 22:33:14.453154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453099 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-socket\") pod \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " Nov 03 22:33:14.453216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453161 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-kube-api-access-hh2cp\" (UniqueName: \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-kube-api-access-hh2cp\") pod \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " Nov 03 22:33:14.453216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453201 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-envoy\") pod \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " Nov 03 22:33:14.453340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453220 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-certs" pod "05f976bf-42fe-4c21-9408-30fc92f0d5bc" (UID: "05f976bf-42fe-4c21-9408-30fc92f0d5bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:14.453340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453234 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-podinfo\") pod \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " Nov 03 22:33:14.453340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453290 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-credential-socket\") pod \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " Nov 03 22:33:14.453340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453334 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-token\" (UniqueName: \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-token\") pod \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " Nov 03 22:33:14.453536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453387 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istiod-ca-cert\") pod \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " Nov 03 22:33:14.453536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453451 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-data\") pod \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\" (UID: \"05f976bf-42fe-4c21-9408-30fc92f0d5bc\") " Nov 03 22:33:14.453713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453381 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-socket" pod "05f976bf-42fe-4c21-9408-30fc92f0d5bc" (UID: "05f976bf-42fe-4c21-9408-30fc92f0d5bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:14.453713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453574 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-credential-socket" pod "05f976bf-42fe-4c21-9408-30fc92f0d5bc" (UID: "05f976bf-42fe-4c21-9408-30fc92f0d5bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:14.454133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.453852 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istiod-ca-cert" pod "05f976bf-42fe-4c21-9408-30fc92f0d5bc" (UID: "05f976bf-42fe-4c21-9408-30fc92f0d5bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:14.454133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.454108 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-data" pod "05f976bf-42fe-4c21-9408-30fc92f0d5bc" (UID: "05f976bf-42fe-4c21-9408-30fc92f0d5bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:14.454489 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.454465 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:14.454565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.454495 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:14.454565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.454511 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:14.454565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.454528 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:14.454565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.454544 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:14.455980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.455954 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-token" pod "05f976bf-42fe-4c21-9408-30fc92f0d5bc" (UID: "05f976bf-42fe-4c21-9408-30fc92f0d5bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:14.456091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.455995 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-kube-api-access-hh2cp" pod "05f976bf-42fe-4c21-9408-30fc92f0d5bc" (UID: "05f976bf-42fe-4c21-9408-30fc92f0d5bc"). InnerVolumeSpecName "kube-api-access-hh2cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:14.456275 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.456253 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-envoy" pod "05f976bf-42fe-4c21-9408-30fc92f0d5bc" (UID: "05f976bf-42fe-4c21-9408-30fc92f0d5bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:14.457073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.457046 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-podinfo" pod "05f976bf-42fe-4c21-9408-30fc92f0d5bc" (UID: "05f976bf-42fe-4c21-9408-30fc92f0d5bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:33:14.509859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.509755 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg"] Nov 03 22:33:14.515795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.515766 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59a07a45-dzkgg"] Nov 03 22:33:14.556324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.556274 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh2cp\" (UniqueName: \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-kube-api-access-hh2cp\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:14.556324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.556315 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:14.556324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.556333 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:14.556712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.556346 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f976bf-42fe-4c21-9408-30fc92f0d5bc-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:14.560546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.560497 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:14.593799 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.593741 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5"] Nov 03 22:33:14.594740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.594059 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" containerID="cri-o://11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1" gracePeriod=30 Nov 03 22:33:14.766681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.766446 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:14.968930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.968882 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:14.970843 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.970812 4298 generic.go:356] "Generic (PLEG): container finished" podID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerID="3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9" exitCode=0 Nov 03 22:33:14.971209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.971188 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" Nov 03 22:33:14.971854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.971828 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" event={"ID":"05f976bf-42fe-4c21-9408-30fc92f0d5bc","Type":"ContainerDied","Data":"3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9"} Nov 03 22:33:14.971962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.971887 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m" event={"ID":"05f976bf-42fe-4c21-9408-30fc92f0d5bc","Type":"ContainerDied","Data":"b5f3e3a4118d59983416627bcc91a162f11386090b33b15ad2c8ad639813bab5"} Nov 03 22:33:14.971962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:14.971923 4298 scope.go:122] "RemoveContainer" containerID="3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9" Nov 03 22:33:15.019514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.019469 4298 scope.go:122] "RemoveContainer" containerID="eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c" Nov 03 22:33:15.042015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.041924 4298 scope.go:122] "RemoveContainer" containerID="3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9" Nov 03 22:33:15.042367 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:33:15.042330 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9\": container with ID starting with 3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9 not found: ID does not exist" containerID="3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9" Nov 03 22:33:15.042429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.042366 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9"} err="failed to get container status \"3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9\": rpc error: code = NotFound desc = could not find container \"3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9\": container with ID starting with 3bdcaec2e9caa202bd0156b4ecb7919da27945ac420982d62459f83e157a65e9 not found: ID does not exist" Nov 03 22:33:15.042429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.042390 4298 scope.go:122] "RemoveContainer" containerID="eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c" Nov 03 22:33:15.042731 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:33:15.042708 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c\": container with ID starting with eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c not found: ID does not exist" containerID="eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c" Nov 03 22:33:15.042794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.042742 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c"} err="failed to get container status \"eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c\": rpc error: code = NotFound desc = could not find container \"eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c\": container with ID starting with eb2069d243760c9bdbe86f37a2fe117eba64a00669086b9f8f7068d9237d760c not found: ID does not exist" Nov 03 22:33:15.153950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.140927 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e212a5a-491d-4410-8189-20b3c53129bd" path="/var/lib/kubelet/pods/6e212a5a-491d-4410-8189-20b3c53129bd/volumes" Nov 03 22:33:15.159617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.159582 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f7c114-f196-4de1-88f3-4878834ee308" path="/var/lib/kubelet/pods/d0f7c114-f196-4de1-88f3-4878834ee308/volumes" Nov 03 22:33:15.204239 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.204212 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m"] Nov 03 22:33:15.225392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.225019 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:15.231061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.231031 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cee4c09d-openshift-default-77f888d699-bgj7m"] Nov 03 22:33:15.386636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.386543 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f73568-8372-4f53-819c-81b7891c60ee-kube-api-access-tr79z\" (UniqueName: \"kubernetes.io/projected/a3f73568-8372-4f53-819c-81b7891c60ee-kube-api-access-tr79z\") pod \"a3f73568-8372-4f53-819c-81b7891c60ee\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " Nov 03 22:33:15.386636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.386617 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-xks-ca\") pod \"a3f73568-8372-4f53-819c-81b7891c60ee\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " Nov 03 22:33:15.386936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.386761 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-openshift-ca\") pod \"a3f73568-8372-4f53-819c-81b7891c60ee\" (UID: \"a3f73568-8372-4f53-819c-81b7891c60ee\") " Nov 03 22:33:15.387187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.387155 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-openshift-ca" pod "a3f73568-8372-4f53-819c-81b7891c60ee" (UID: "a3f73568-8372-4f53-819c-81b7891c60ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:15.387566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.387542 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:15.389538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.389300 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f73568-8372-4f53-819c-81b7891c60ee-kube-api-access-tr79z" pod "a3f73568-8372-4f53-819c-81b7891c60ee" (UID: "a3f73568-8372-4f53-819c-81b7891c60ee"). InnerVolumeSpecName "kube-api-access-tr79z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:15.391108 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.391080 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-xks-ca" pod "a3f73568-8372-4f53-819c-81b7891c60ee" (UID: "a3f73568-8372-4f53-819c-81b7891c60ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:15.488656 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.488610 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr79z\" (UniqueName: \"kubernetes.io/projected/a3f73568-8372-4f53-819c-81b7891c60ee-kube-api-access-tr79z\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:15.488855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.488689 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3f73568-8372-4f53-819c-81b7891c60ee-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:15.561465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.561408 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:15.984228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.984196 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" Nov 03 22:33:15.984637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.984234 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj" event={"ID":"a3f73568-8372-4f53-819c-81b7891c60ee","Type":"ContainerDied","Data":"4d868d438dff211a13cb673e4d055ed782e59e41e1353823149c8dd3654c0e5a"} Nov 03 22:33:15.984637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:15.984279 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d868d438dff211a13cb673e4d055ed782e59e41e1353823149c8dd3654c0e5a" Nov 03 22:33:16.262778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:16.221141 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:16.502552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:16.502509 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:17.042845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:17.042808 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:17.136537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:17.136505 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" path="/var/lib/kubelet/pods/05f976bf-42fe-4c21-9408-30fc92f0d5bc/volumes" Nov 03 22:33:17.726554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:17.726507 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:17.767951 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:33:17.767914 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:33:18.023567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:18.023471 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:18.057368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:18.057335 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb"] Nov 03 22:33:18.106259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:18.106225 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56417af0-vfztb"] Nov 03 22:33:19.135810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.135780 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d179689f-e41e-4d6d-b6b2-8aafe41e5c7b" path="/var/lib/kubelet/pods/d179689f-e41e-4d6d-b6b2-8aafe41e5c7b/volumes" Nov 03 22:33:19.531934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.531901 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:19.922688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.922655 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:33:19.976524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.976492 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-token\" (UniqueName: \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-token\") pod \"d70853f4-2318-4175-9e48-9c8e2b7e0439\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " Nov 03 22:33:19.976713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.976598 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-socket\") pod \"d70853f4-2318-4175-9e48-9c8e2b7e0439\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " Nov 03 22:33:19.976713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.976674 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70853f4-2318-4175-9e48-9c8e2b7e0439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70853f4-2318-4175-9e48-9c8e2b7e0439-istiod-ca-cert\") pod \"d70853f4-2318-4175-9e48-9c8e2b7e0439\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " Nov 03 22:33:19.976820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.976739 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-certs\") pod \"d70853f4-2318-4175-9e48-9c8e2b7e0439\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " Nov 03 22:33:19.976820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.976779 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-envoy\") pod \"d70853f4-2318-4175-9e48-9c8e2b7e0439\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " Nov 03 22:33:19.976967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.976947 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-credential-socket\") pod \"d70853f4-2318-4175-9e48-9c8e2b7e0439\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " Nov 03 22:33:19.977032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.976997 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-kube-api-access-g2xtj\" (UniqueName: \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-kube-api-access-g2xtj\") pod \"d70853f4-2318-4175-9e48-9c8e2b7e0439\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " Nov 03 22:33:19.977032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.976941 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-socket" pod "d70853f4-2318-4175-9e48-9c8e2b7e0439" (UID: "d70853f4-2318-4175-9e48-9c8e2b7e0439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:19.977142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.977060 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-podinfo\") pod \"d70853f4-2318-4175-9e48-9c8e2b7e0439\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " Nov 03 22:33:19.977142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.977081 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-certs" pod "d70853f4-2318-4175-9e48-9c8e2b7e0439" (UID: "d70853f4-2318-4175-9e48-9c8e2b7e0439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:19.977260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.977153 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-credential-socket" pod "d70853f4-2318-4175-9e48-9c8e2b7e0439" (UID: "d70853f4-2318-4175-9e48-9c8e2b7e0439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:19.977260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.977169 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-data\") pod \"d70853f4-2318-4175-9e48-9c8e2b7e0439\" (UID: \"d70853f4-2318-4175-9e48-9c8e2b7e0439\") " Nov 03 22:33:19.977469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.977448 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-data" pod "d70853f4-2318-4175-9e48-9c8e2b7e0439" (UID: "d70853f4-2318-4175-9e48-9c8e2b7e0439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:19.977543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.977468 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70853f4-2318-4175-9e48-9c8e2b7e0439-istiod-ca-cert" pod "d70853f4-2318-4175-9e48-9c8e2b7e0439" (UID: "d70853f4-2318-4175-9e48-9c8e2b7e0439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:19.978109 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.978085 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:19.978221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.978114 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:19.978221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.978128 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:19.978221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.978142 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:19.978221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.978157 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70853f4-2318-4175-9e48-9c8e2b7e0439-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:19.979435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.979408 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-token" pod "d70853f4-2318-4175-9e48-9c8e2b7e0439" (UID: "d70853f4-2318-4175-9e48-9c8e2b7e0439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:19.979684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.979630 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-podinfo" pod "d70853f4-2318-4175-9e48-9c8e2b7e0439" (UID: "d70853f4-2318-4175-9e48-9c8e2b7e0439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:33:19.979684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.979633 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-envoy" pod "d70853f4-2318-4175-9e48-9c8e2b7e0439" (UID: "d70853f4-2318-4175-9e48-9c8e2b7e0439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:19.980334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:19.980314 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-kube-api-access-g2xtj" pod "d70853f4-2318-4175-9e48-9c8e2b7e0439" (UID: "d70853f4-2318-4175-9e48-9c8e2b7e0439"). InnerVolumeSpecName "kube-api-access-g2xtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:20.021059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.021025 4298 generic.go:356] "Generic (PLEG): container finished" podID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerID="11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1" exitCode=0 Nov 03 22:33:20.021228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.021097 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" Nov 03 22:33:20.021228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.021116 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" event={"ID":"d70853f4-2318-4175-9e48-9c8e2b7e0439","Type":"ContainerDied","Data":"11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1"} Nov 03 22:33:20.021228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.021186 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5" event={"ID":"d70853f4-2318-4175-9e48-9c8e2b7e0439","Type":"ContainerDied","Data":"4fe72a6af5d47aa479fa1d95c309da621791289ac93a35ed1947d2302b343018"} Nov 03 22:33:20.021228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.021215 4298 scope.go:122] "RemoveContainer" containerID="11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1" Nov 03 22:33:20.034589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.034572 4298 scope.go:122] "RemoveContainer" containerID="44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc" Nov 03 22:33:20.046855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.046830 4298 scope.go:122] "RemoveContainer" containerID="11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1" Nov 03 22:33:20.047177 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:33:20.047157 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1\": container with ID starting with 11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1 not found: ID does not exist" containerID="11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1" Nov 03 22:33:20.047242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.047184 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1"} err="failed to get container status \"11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1\": rpc error: code = NotFound desc = could not find container \"11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1\": container with ID starting with 11174a46724d2a6eddaa2200b0b195c2b359c1fec1267cd1c217c27d83d6f7a1 not found: ID does not exist" Nov 03 22:33:20.047242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.047203 4298 scope.go:122] "RemoveContainer" containerID="44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc" Nov 03 22:33:20.047532 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:33:20.047499 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc\": container with ID starting with 44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc not found: ID does not exist" containerID="44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc" Nov 03 22:33:20.047583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.047541 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc"} err="failed to get container status \"44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc\": rpc error: code = NotFound desc = could not find container \"44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc\": container with ID starting with 44dc96bb8190ec570d74b523a56c1f5aae4e8a92a522145d4bc193745f1f65dc not found: ID does not exist" Nov 03 22:33:20.080737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.080708 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.080840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.080741 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.080840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.080760 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70853f4-2318-4175-9e48-9c8e2b7e0439-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.080840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.080777 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2xtj\" (UniqueName: \"kubernetes.io/projected/d70853f4-2318-4175-9e48-9c8e2b7e0439-kube-api-access-g2xtj\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.262269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.262228 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5"] Nov 03 22:33:20.268240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.268211 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-59a07a45-gw-openshift-default-d947cddd9-nzjw5"] Nov 03 22:33:20.322582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.322490 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:20.508182 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.508141 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:20.698939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:20.698904 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:21.136859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.136786 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" path="/var/lib/kubelet/pods/d70853f4-2318-4175-9e48-9c8e2b7e0439/volumes" Nov 03 22:33:21.950188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.950155 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j"] Nov 03 22:33:21.950836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.950821 4298 cpu_manager.go:414] "containerMap: removing container" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" Nov 03 22:33:21.950895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.950838 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" Nov 03 22:33:21.950895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.950851 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" Nov 03 22:33:21.950895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.950856 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" Nov 03 22:33:21.950895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.950868 4298 cpu_manager.go:414] "containerMap: removing container" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" Nov 03 22:33:21.950895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.950872 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" Nov 03 22:33:21.950895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.950893 4298 cpu_manager.go:414] "containerMap: removing container" podUID="a3f73568-8372-4f53-819c-81b7891c60ee" containerName="revoke-keys" Nov 03 22:33:21.951080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.950898 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f73568-8372-4f53-819c-81b7891c60ee" containerName="revoke-keys" Nov 03 22:33:21.951080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.951025 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" Nov 03 22:33:21.951080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.951034 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" Nov 03 22:33:21.951080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.951041 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" Nov 03 22:33:21.951080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.951047 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f976bf-42fe-4c21-9408-30fc92f0d5bc" containerName="istio-proxy" Nov 03 22:33:21.951080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.951065 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f73568-8372-4f53-819c-81b7891c60ee" containerName="revoke-keys" Nov 03 22:33:21.956433 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:21.956412 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:22.008299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.008261 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248efb3a-7ss4j\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:22.008496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.008427 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248efb3a-7ss4j\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:22.008691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.008673 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlgdl\" (UniqueName: \"kubernetes.io/projected/d106648c-97a0-41e5-8c03-3d788bca08cb-kube-api-access-dlgdl\") pod \"maas-api-revoke-sub-248efb3a-7ss4j\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:22.009538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.009516 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j"] Nov 03 22:33:22.072794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.072763 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:33:22.072794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.072798 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:33:22.110272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.110237 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlgdl\" (UniqueName: \"kubernetes.io/projected/d106648c-97a0-41e5-8c03-3d788bca08cb-kube-api-access-dlgdl\") pod \"maas-api-revoke-sub-248efb3a-7ss4j\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:22.110438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.110293 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248efb3a-7ss4j\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:22.110438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.110372 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248efb3a-7ss4j\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:22.111050 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.111025 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248efb3a-7ss4j\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:22.112631 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.112608 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248efb3a-7ss4j\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:22.167095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.167064 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlgdl\" (UniqueName: \"kubernetes.io/projected/d106648c-97a0-41e5-8c03-3d788bca08cb-kube-api-access-dlgdl\") pod \"maas-api-revoke-sub-248efb3a-7ss4j\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:22.271880 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.271798 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:22.646209 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:33:22.646172 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd106648c_97a0_41e5_8c03_3d788bca08cb.slice/crio-26db6aca36838ca75d6319fcdde1e60debead7cd48e5802c057dba0605b59f44 WatchSource:0}: Error finding container 26db6aca36838ca75d6319fcdde1e60debead7cd48e5802c057dba0605b59f44: Status 404 returned error can't find the container with id 26db6aca36838ca75d6319fcdde1e60debead7cd48e5802c057dba0605b59f44 Nov 03 22:33:22.652052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:22.652031 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j"] Nov 03 22:33:23.010466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:23.010434 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4"] Nov 03 22:33:23.047669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:23.047620 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" event={"ID":"d106648c-97a0-41e5-8c03-3d788bca08cb","Type":"ContainerStarted","Data":"eb8a3e9f55e18602dfdbb0b113990e4f08c09fb3f4165cc61220e6303183bf51"} Nov 03 22:33:23.047899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:23.047693 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" event={"ID":"d106648c-97a0-41e5-8c03-3d788bca08cb","Type":"ContainerStarted","Data":"26db6aca36838ca75d6319fcdde1e60debead7cd48e5802c057dba0605b59f44"} Nov 03 22:33:23.192918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:23.192794 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:23.284117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:23.284037 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" podStartSLOduration=2.284015931 podStartE2EDuration="2.284015931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:23.273086465 +0000 UTC m=+1560.253820118" watchObservedRunningTime="2026-11-03 22:33:23.284015931 +0000 UTC m=+1560.264749589" Nov 03 22:33:23.384712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:23.384628 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:23.599475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:23.599433 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:24.057176 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:24.057137 4298 generic.go:356] "Generic (PLEG): container finished" podID="d106648c-97a0-41e5-8c03-3d788bca08cb" containerID="eb8a3e9f55e18602dfdbb0b113990e4f08c09fb3f4165cc61220e6303183bf51" exitCode=0 Nov 03 22:33:24.057655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:24.057221 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" event={"ID":"d106648c-97a0-41e5-8c03-3d788bca08cb","Type":"ContainerDied","Data":"eb8a3e9f55e18602dfdbb0b113990e4f08c09fb3f4165cc61220e6303183bf51"} Nov 03 22:33:24.432417 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:24.432323 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:24.707949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:24.707902 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:25.060167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.060076 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk"] Nov 03 22:33:25.108230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.108164 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a292514-24bnk"] Nov 03 22:33:25.163161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.160474 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0107e812-54d9-4285-af97-7e41db1f17fc" path="/var/lib/kubelet/pods/0107e812-54d9-4285-af97-7e41db1f17fc/volumes" Nov 03 22:33:25.326364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.326298 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:25.371824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.371787 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d106648c-97a0-41e5-8c03-3d788bca08cb-kube-api-access-dlgdl\" (UniqueName: \"kubernetes.io/projected/d106648c-97a0-41e5-8c03-3d788bca08cb-kube-api-access-dlgdl\") pod \"d106648c-97a0-41e5-8c03-3d788bca08cb\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " Nov 03 22:33:25.372008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.371883 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-openshift-ca\") pod \"d106648c-97a0-41e5-8c03-3d788bca08cb\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " Nov 03 22:33:25.372119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.372098 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-xks-ca\") pod \"d106648c-97a0-41e5-8c03-3d788bca08cb\" (UID: \"d106648c-97a0-41e5-8c03-3d788bca08cb\") " Nov 03 22:33:25.372275 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.372250 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-openshift-ca" pod "d106648c-97a0-41e5-8c03-3d788bca08cb" (UID: "d106648c-97a0-41e5-8c03-3d788bca08cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:25.372714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.372698 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:25.374007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.373978 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-xks-ca" pod "d106648c-97a0-41e5-8c03-3d788bca08cb" (UID: "d106648c-97a0-41e5-8c03-3d788bca08cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:25.374130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.374007 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d106648c-97a0-41e5-8c03-3d788bca08cb-kube-api-access-dlgdl" pod "d106648c-97a0-41e5-8c03-3d788bca08cb" (UID: "d106648c-97a0-41e5-8c03-3d788bca08cb"). InnerVolumeSpecName "kube-api-access-dlgdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:25.474359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.474318 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlgdl\" (UniqueName: \"kubernetes.io/projected/d106648c-97a0-41e5-8c03-3d788bca08cb-kube-api-access-dlgdl\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:25.474359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:25.474356 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d106648c-97a0-41e5-8c03-3d788bca08cb-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:26.075329 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:26.075299 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" Nov 03 22:33:26.075776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:26.075341 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j" event={"ID":"d106648c-97a0-41e5-8c03-3d788bca08cb","Type":"ContainerDied","Data":"26db6aca36838ca75d6319fcdde1e60debead7cd48e5802c057dba0605b59f44"} Nov 03 22:33:26.075776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:26.075404 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26db6aca36838ca75d6319fcdde1e60debead7cd48e5802c057dba0605b59f44" Nov 03 22:33:27.132723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:27.132690 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:27.311619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:27.311578 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:28.011468 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:28.011424 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" podUID="c497ebcf-a920-40cd-a477-38ac0eeb6350" containerName="payload-pre-processing" containerID="cri-o://a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932" gracePeriod=25 Nov 03 22:33:28.319085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:28.319056 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:33:28.425701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:28.425626 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c497ebcf-a920-40cd-a477-38ac0eeb6350-kube-api-access-frl8v\" (UniqueName: \"kubernetes.io/projected/c497ebcf-a920-40cd-a477-38ac0eeb6350-kube-api-access-frl8v\") pod \"c497ebcf-a920-40cd-a477-38ac0eeb6350\" (UID: \"c497ebcf-a920-40cd-a477-38ac0eeb6350\") " Nov 03 22:33:28.425874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:28.425812 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c497ebcf-a920-40cd-a477-38ac0eeb6350-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c497ebcf-a920-40cd-a477-38ac0eeb6350-plugins-config-volume\") pod \"c497ebcf-a920-40cd-a477-38ac0eeb6350\" (UID: \"c497ebcf-a920-40cd-a477-38ac0eeb6350\") " Nov 03 22:33:28.426231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:28.426203 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c497ebcf-a920-40cd-a477-38ac0eeb6350-plugins-config-volume" pod "c497ebcf-a920-40cd-a477-38ac0eeb6350" (UID: "c497ebcf-a920-40cd-a477-38ac0eeb6350"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:28.426507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:28.426490 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c497ebcf-a920-40cd-a477-38ac0eeb6350-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:28.427889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:28.427858 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c497ebcf-a920-40cd-a477-38ac0eeb6350-kube-api-access-frl8v" pod "c497ebcf-a920-40cd-a477-38ac0eeb6350" (UID: "c497ebcf-a920-40cd-a477-38ac0eeb6350"). InnerVolumeSpecName "kube-api-access-frl8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:28.528043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:28.528005 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frl8v\" (UniqueName: \"kubernetes.io/projected/c497ebcf-a920-40cd-a477-38ac0eeb6350-kube-api-access-frl8v\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:29.101211 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:29.101180 4298 generic.go:356] "Generic (PLEG): container finished" podID="c497ebcf-a920-40cd-a477-38ac0eeb6350" containerID="a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932" exitCode=0 Nov 03 22:33:29.101427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:29.101236 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" Nov 03 22:33:29.101427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:29.101267 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" event={"ID":"c497ebcf-a920-40cd-a477-38ac0eeb6350","Type":"ContainerDied","Data":"a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932"} Nov 03 22:33:29.101427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:29.101329 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4" event={"ID":"c497ebcf-a920-40cd-a477-38ac0eeb6350","Type":"ContainerDied","Data":"b9d2bbd3db7230bbf05cd911026c48aadda68a0362009267fdfb0e384450557a"} Nov 03 22:33:29.101427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:29.101351 4298 scope.go:122] "RemoveContainer" containerID="a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932" Nov 03 22:33:29.113426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:29.113403 4298 scope.go:122] "RemoveContainer" containerID="a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932" Nov 03 22:33:29.113781 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:33:29.113760 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932\": container with ID starting with a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932 not found: ID does not exist" containerID="a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932" Nov 03 22:33:29.113848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:29.113794 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932"} err="failed to get container status \"a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932\": rpc error: code = NotFound desc = could not find container \"a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932\": container with ID starting with a31aeeed0395c779ded3dc43d77b51ddcb0d347013c9a54abf21bdfa24b4d932 not found: ID does not exist" Nov 03 22:33:29.270031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:29.269997 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4"] Nov 03 22:33:29.320574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:29.320538 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1c72fb7-5f67bd4b4c-4sgs4"] Nov 03 22:33:29.583796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:29.583760 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:30.019123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:30.019094 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:30.234730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:30.234638 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:31.136321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:31.136289 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c497ebcf-a920-40cd-a477-38ac0eeb6350" path="/var/lib/kubelet/pods/c497ebcf-a920-40cd-a477-38ac0eeb6350/volumes" Nov 03 22:33:32.052305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:32.052265 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd"] Nov 03 22:33:32.101693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:32.101657 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf6da90f-z58pd"] Nov 03 22:33:32.638638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:32.638601 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:33.137278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:33.137240 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18f12da7-4753-4bee-ae76-74ac30f892f9" path="/var/lib/kubelet/pods/18f12da7-4753-4bee-ae76-74ac30f892f9/volumes" Nov 03 22:33:34.186142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:34.186111 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:35.553837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:35.553805 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:38.272395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.272351 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:38.541260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.541179 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb"] Nov 03 22:33:38.541856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.541841 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" Nov 03 22:33:38.541904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.541856 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70853f4-2318-4175-9e48-9c8e2b7e0439" containerName="istio-proxy" Nov 03 22:33:38.541904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.541885 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d106648c-97a0-41e5-8c03-3d788bca08cb" containerName="revoke-keys" Nov 03 22:33:38.541904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.541891 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d106648c-97a0-41e5-8c03-3d788bca08cb" containerName="revoke-keys" Nov 03 22:33:38.542012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.541915 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c497ebcf-a920-40cd-a477-38ac0eeb6350" containerName="payload-pre-processing" Nov 03 22:33:38.542012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.541921 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c497ebcf-a920-40cd-a477-38ac0eeb6350" containerName="payload-pre-processing" Nov 03 22:33:38.542075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.542023 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d106648c-97a0-41e5-8c03-3d788bca08cb" containerName="revoke-keys" Nov 03 22:33:38.542075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.542036 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c497ebcf-a920-40cd-a477-38ac0eeb6350" containerName="payload-pre-processing" Nov 03 22:33:38.546920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.546896 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.599588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.599551 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 22:33:38.599588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.599551 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 22:33:38.599872 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.599620 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 22:33:38.599872 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.599675 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qcrf7\"" Nov 03 22:33:38.656617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.656583 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb"] Nov 03 22:33:38.689688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.689469 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.689688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.689534 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.689688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.689588 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.689688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.689653 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.690000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.689749 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/948f4945-f3f1-4308-b010-846ab3a4616b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.690000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.689781 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.690000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.689821 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwfb\" (UniqueName: \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-kube-api-access-ptwfb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.791009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.790976 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.791212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.791080 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/948f4945-f3f1-4308-b010-846ab3a4616b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.791212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.791111 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.791212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.791150 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwfb\" (UniqueName: \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-kube-api-access-ptwfb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.791437 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.791391 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.791501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.791440 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.791501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.791495 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.792313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.792284 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.793788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.793763 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.793788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.793780 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/948f4945-f3f1-4308-b010-846ab3a4616b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.793922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.793807 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.793975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.793953 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.803484 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.803457 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:38.851717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.851689 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.851940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.851919 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwfb\" (UniqueName: \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-kube-api-access-ptwfb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcncb\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:38.860553 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:38.860531 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:39.285718 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:33:39.285689 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948f4945_f3f1_4308_b010_846ab3a4616b.slice/crio-43a5255078c484268528477577d43484f3053e3e906ec1ba07c62b00b3b75778 WatchSource:0}: Error finding container 43a5255078c484268528477577d43484f3053e3e906ec1ba07c62b00b3b75778: Status 404 returned error can't find the container with id 43a5255078c484268528477577d43484f3053e3e906ec1ba07c62b00b3b75778 Nov 03 22:33:39.288004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:39.287986 4298 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:33:39.291923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:39.291895 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb"] Nov 03 22:33:39.294918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:39.294898 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" event={"ID":"948f4945-f3f1-4308-b010-846ab3a4616b","Type":"ContainerStarted","Data":"43a5255078c484268528477577d43484f3053e3e906ec1ba07c62b00b3b75778"} Nov 03 22:33:41.242179 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:41.242144 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:42.643754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:42.643714 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:43.266945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:43.266914 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:33:43.267019 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:43.266976 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:33:44.243879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:44.243843 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:44.342554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:44.342492 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" event={"ID":"948f4945-f3f1-4308-b010-846ab3a4616b","Type":"ContainerStarted","Data":"afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317"} Nov 03 22:33:44.342786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:44.342591 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:44.344981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:44.344941 4298 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dcncb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:33:44.345110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:44.345020 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" podUID="948f4945-f3f1-4308-b010-846ab3a4616b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:33:44.543526 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:44.543396 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" podStartSLOduration=2.565976353 podStartE2EDuration="6.543370194s" totalImagesPullingTime="3.977393841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:44.520802705 +0000 UTC m=+1581.501536359" watchObservedRunningTime="2026-11-03 22:33:44.543370194 +0000 UTC m=+1581.524103848" Nov 03 22:33:44.613557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:44.613520 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:44.798253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:44.798115 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj"] Nov 03 22:33:44.810675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:44.810485 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:44.840751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:44.840712 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1c72fb7-pzcnj"] Nov 03 22:33:45.136323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:45.136248 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f73568-8372-4f53-819c-81b7891c60ee" path="/var/lib/kubelet/pods/a3f73568-8372-4f53-819c-81b7891c60ee/volumes" Nov 03 22:33:45.165121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:45.165086 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:45.350377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:45.350349 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:33:47.553844 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:33:47.553806 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:33:49.050137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:49.050103 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct"] Nov 03 22:33:49.056106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:49.056076 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c18b0ea2-gsdct"] Nov 03 22:33:49.135989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:49.135960 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca3f8ad-3bf1-4aa9-9b87-093405c5536c" path="/var/lib/kubelet/pods/8ca3f8ad-3bf1-4aa9-9b87-093405c5536c/volumes" Nov 03 22:33:49.400619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:49.400527 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:49.579516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:49.579478 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:50.785966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:50.785931 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:50.961578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:50.961550 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:51.063394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:51.063294 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv"] Nov 03 22:33:51.063626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:51.063595 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" containerID="cri-o://f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438" gracePeriod=30 Nov 03 22:33:51.203666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:51.193543 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:51.585814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:51.585780 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:51.773357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:51.773329 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:52.007918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:52.007876 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:54.091629 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:54.091593 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:54.093711 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:54.093680 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg"] Nov 03 22:33:54.116899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:54.116859 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5d667a-7hvlg"] Nov 03 22:33:55.136908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:55.136873 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e58db0-fee5-4007-a5ba-0c7f181d7f08" path="/var/lib/kubelet/pods/66e58db0-fee5-4007-a5ba-0c7f181d7f08/volumes" Nov 03 22:33:56.368362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.368335 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:33:56.440067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.440032 4298 generic.go:356] "Generic (PLEG): container finished" podID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerID="f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438" exitCode=0 Nov 03 22:33:56.440266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.440074 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" event={"ID":"221e35d0-71d1-4488-8108-6dcb28e6b621","Type":"ContainerDied","Data":"f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438"} Nov 03 22:33:56.440266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.440114 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" event={"ID":"221e35d0-71d1-4488-8108-6dcb28e6b621","Type":"ContainerDied","Data":"366501e8784016bc0332d748c0d047b85430167edce8a9d19df54dcfb8c6fb97"} Nov 03 22:33:56.440266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.440137 4298 scope.go:122] "RemoveContainer" containerID="f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438" Nov 03 22:33:56.440266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.440079 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv" Nov 03 22:33:56.440933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.440911 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-kube-api-access-86xz2\" (UniqueName: \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-kube-api-access-86xz2\") pod \"221e35d0-71d1-4488-8108-6dcb28e6b621\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " Nov 03 22:33:56.441064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.440950 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-envoy\") pod \"221e35d0-71d1-4488-8108-6dcb28e6b621\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " Nov 03 22:33:56.441064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.441014 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-socket\") pod \"221e35d0-71d1-4488-8108-6dcb28e6b621\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " Nov 03 22:33:56.441189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.441110 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-token\" (UniqueName: \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-token\") pod \"221e35d0-71d1-4488-8108-6dcb28e6b621\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " Nov 03 22:33:56.441189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.441175 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221e35d0-71d1-4488-8108-6dcb28e6b621-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221e35d0-71d1-4488-8108-6dcb28e6b621-istiod-ca-cert\") pod \"221e35d0-71d1-4488-8108-6dcb28e6b621\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " Nov 03 22:33:56.441287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.441210 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-data\") pod \"221e35d0-71d1-4488-8108-6dcb28e6b621\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " Nov 03 22:33:56.441339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.441305 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-certs\") pod \"221e35d0-71d1-4488-8108-6dcb28e6b621\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " Nov 03 22:33:56.441599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.441406 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-podinfo\") pod \"221e35d0-71d1-4488-8108-6dcb28e6b621\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " Nov 03 22:33:56.441599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.441497 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-credential-socket\") pod \"221e35d0-71d1-4488-8108-6dcb28e6b621\" (UID: \"221e35d0-71d1-4488-8108-6dcb28e6b621\") " Nov 03 22:33:56.442096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.441414 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-socket" pod "221e35d0-71d1-4488-8108-6dcb28e6b621" (UID: "221e35d0-71d1-4488-8108-6dcb28e6b621"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:56.442279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.442257 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-credential-socket" pod "221e35d0-71d1-4488-8108-6dcb28e6b621" (UID: "221e35d0-71d1-4488-8108-6dcb28e6b621"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:56.442279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.441978 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221e35d0-71d1-4488-8108-6dcb28e6b621-istiod-ca-cert" pod "221e35d0-71d1-4488-8108-6dcb28e6b621" (UID: "221e35d0-71d1-4488-8108-6dcb28e6b621"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:56.442448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.442428 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:56.442543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.442449 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221e35d0-71d1-4488-8108-6dcb28e6b621-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:56.442543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.442462 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:56.443062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.442902 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-data" pod "221e35d0-71d1-4488-8108-6dcb28e6b621" (UID: "221e35d0-71d1-4488-8108-6dcb28e6b621"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:56.443062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.442978 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-certs" pod "221e35d0-71d1-4488-8108-6dcb28e6b621" (UID: "221e35d0-71d1-4488-8108-6dcb28e6b621"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:56.443584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.443554 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-kube-api-access-86xz2" pod "221e35d0-71d1-4488-8108-6dcb28e6b621" (UID: "221e35d0-71d1-4488-8108-6dcb28e6b621"). InnerVolumeSpecName "kube-api-access-86xz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:56.443874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.443852 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-token" pod "221e35d0-71d1-4488-8108-6dcb28e6b621" (UID: "221e35d0-71d1-4488-8108-6dcb28e6b621"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:56.443962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.443851 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-envoy" pod "221e35d0-71d1-4488-8108-6dcb28e6b621" (UID: "221e35d0-71d1-4488-8108-6dcb28e6b621"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:56.445229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.445198 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-podinfo" pod "221e35d0-71d1-4488-8108-6dcb28e6b621" (UID: "221e35d0-71d1-4488-8108-6dcb28e6b621"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:33:56.475099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.475076 4298 scope.go:122] "RemoveContainer" containerID="b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3" Nov 03 22:33:56.485540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.485518 4298 scope.go:122] "RemoveContainer" containerID="f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438" Nov 03 22:33:56.485945 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:33:56.485909 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438\": container with ID starting with f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438 not found: ID does not exist" containerID="f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438" Nov 03 22:33:56.486061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.485957 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438"} err="failed to get container status \"f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438\": rpc error: code = NotFound desc = could not find container \"f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438\": container with ID starting with f3e59e0998fe476c35efb8da08f030d5fa028b8e183266ca05dff8199a0da438 not found: ID does not exist" Nov 03 22:33:56.486061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.485990 4298 scope.go:122] "RemoveContainer" containerID="b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3" Nov 03 22:33:56.486306 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:33:56.486271 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3\": container with ID starting with b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3 not found: ID does not exist" containerID="b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3" Nov 03 22:33:56.486393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.486316 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3"} err="failed to get container status \"b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3\": rpc error: code = NotFound desc = could not find container \"b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3\": container with ID starting with b0b21b009fe03ee9f686489a80bc0187fc5bc94568ee60495ebd65311ca5adf3 not found: ID does not exist" Nov 03 22:33:56.544590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.544560 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:56.544590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.544588 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86xz2\" (UniqueName: \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-kube-api-access-86xz2\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:56.544590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.544598 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:56.544854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.544606 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:56.544854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.544614 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:56.544854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.544623 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221e35d0-71d1-4488-8108-6dcb28e6b621-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:56.868389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.868347 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv"] Nov 03 22:33:56.875039 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:56.875010 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e1c72fb7-openshift-default-5f6848d6cc-nsbdv"] Nov 03 22:33:57.136599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:57.136521 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" path="/var/lib/kubelet/pods/221e35d0-71d1-4488-8108-6dcb28e6b621/volumes" Nov 03 22:33:57.613128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:57.613094 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:57.791432 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:57.791399 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:58.212359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:58.212316 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:58.398342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:58.398256 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:58.583408 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:58.583317 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:58.774421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:58.774371 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:59.540128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:59.540096 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:59.726170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:33:59.726139 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:00.064340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:00.064309 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x"] Nov 03 22:34:00.114604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:00.114569 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c933eb6-6d64x"] Nov 03 22:34:01.136258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:01.136225 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced587dd-6d32-40c2-816e-e6c9208d356f" path="/var/lib/kubelet/pods/ced587dd-6d32-40c2-816e-e6c9208d356f/volumes" Nov 03 22:34:03.374179 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:03.374140 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:06.560030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.559998 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc"] Nov 03 22:34:06.560668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.560632 4298 cpu_manager.go:414] "containerMap: removing container" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" Nov 03 22:34:06.560668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.560658 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" Nov 03 22:34:06.560793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.560784 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" Nov 03 22:34:06.560829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.560802 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" Nov 03 22:34:06.566059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.566035 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:06.601733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.601697 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3174043-6j8xc\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:06.601915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.601884 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxv5\" (UniqueName: \"kubernetes.io/projected/8ab391ca-993d-40cf-bf77-b5ed2618951d-kube-api-access-fmxv5\") pod \"maas-api-revoke-sub-b3174043-6j8xc\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:06.601966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.601940 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3174043-6j8xc\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:06.617755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.617722 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc"] Nov 03 22:34:06.660270 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.660233 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:34:06.660456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.660310 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:34:06.703880 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.703846 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3174043-6j8xc\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:06.704078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.703988 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxv5\" (UniqueName: \"kubernetes.io/projected/8ab391ca-993d-40cf-bf77-b5ed2618951d-kube-api-access-fmxv5\") pod \"maas-api-revoke-sub-b3174043-6j8xc\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:06.704078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.704036 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3174043-6j8xc\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:06.704660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.704615 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3174043-6j8xc\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:06.706247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.706221 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3174043-6j8xc\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:06.761063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.761031 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxv5\" (UniqueName: \"kubernetes.io/projected/8ab391ca-993d-40cf-bf77-b5ed2618951d-kube-api-access-fmxv5\") pod \"maas-api-revoke-sub-b3174043-6j8xc\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:06.880309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:06.880214 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:07.257792 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:34:07.257757 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab391ca_993d_40cf_bf77_b5ed2618951d.slice/crio-d08109a7f3feba0396e190f4f85dd6d7fcf612bbcfd4b14b8c094c347b715ef6 WatchSource:0}: Error finding container d08109a7f3feba0396e190f4f85dd6d7fcf612bbcfd4b14b8c094c347b715ef6: Status 404 returned error can't find the container with id d08109a7f3feba0396e190f4f85dd6d7fcf612bbcfd4b14b8c094c347b715ef6 Nov 03 22:34:07.263373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:07.263346 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc"] Nov 03 22:34:07.549767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:07.549735 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" event={"ID":"8ab391ca-993d-40cf-bf77-b5ed2618951d","Type":"ContainerStarted","Data":"97eb309c417676a1230761c070d3c8aece53712a6781f51ea5a5cc1ce492ec53"} Nov 03 22:34:07.549935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:07.549783 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" event={"ID":"8ab391ca-993d-40cf-bf77-b5ed2618951d","Type":"ContainerStarted","Data":"d08109a7f3feba0396e190f4f85dd6d7fcf612bbcfd4b14b8c094c347b715ef6"} Nov 03 22:34:07.668084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:07.668028 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" podStartSLOduration=1.6680130210000002 podStartE2EDuration="1.668013021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:07.66238118 +0000 UTC m=+1604.643114833" watchObservedRunningTime="2026-11-03 22:34:07.668013021 +0000 UTC m=+1604.648746675" Nov 03 22:34:07.767266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:07.767227 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:07.950888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:07.950850 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:08.129189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:08.129155 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:08.558970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:08.558877 4298 generic.go:356] "Generic (PLEG): container finished" podID="8ab391ca-993d-40cf-bf77-b5ed2618951d" containerID="97eb309c417676a1230761c070d3c8aece53712a6781f51ea5a5cc1ce492ec53" exitCode=0 Nov 03 22:34:08.559135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:08.558958 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" event={"ID":"8ab391ca-993d-40cf-bf77-b5ed2618951d","Type":"ContainerDied","Data":"97eb309c417676a1230761c070d3c8aece53712a6781f51ea5a5cc1ce492ec53"} Nov 03 22:34:09.784259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:09.784228 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:09.851224 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:09.851137 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-openshift-ca\") pod \"8ab391ca-993d-40cf-bf77-b5ed2618951d\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " Nov 03 22:34:09.851423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:09.851399 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab391ca-993d-40cf-bf77-b5ed2618951d-kube-api-access-fmxv5\" (UniqueName: \"kubernetes.io/projected/8ab391ca-993d-40cf-bf77-b5ed2618951d-kube-api-access-fmxv5\") pod \"8ab391ca-993d-40cf-bf77-b5ed2618951d\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " Nov 03 22:34:09.851544 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:09.851451 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-openshift-ca" pod "8ab391ca-993d-40cf-bf77-b5ed2618951d" (UID: "8ab391ca-993d-40cf-bf77-b5ed2618951d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:34:09.851544 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:09.851506 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-xks-ca\") pod \"8ab391ca-993d-40cf-bf77-b5ed2618951d\" (UID: \"8ab391ca-993d-40cf-bf77-b5ed2618951d\") " Nov 03 22:34:09.852165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:09.852141 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:34:09.853507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:09.853477 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-xks-ca" pod "8ab391ca-993d-40cf-bf77-b5ed2618951d" (UID: "8ab391ca-993d-40cf-bf77-b5ed2618951d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:34:09.853690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:09.853621 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab391ca-993d-40cf-bf77-b5ed2618951d-kube-api-access-fmxv5" pod "8ab391ca-993d-40cf-bf77-b5ed2618951d" (UID: "8ab391ca-993d-40cf-bf77-b5ed2618951d"). InnerVolumeSpecName "kube-api-access-fmxv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:09.953739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:09.953703 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmxv5\" (UniqueName: \"kubernetes.io/projected/8ab391ca-993d-40cf-bf77-b5ed2618951d-kube-api-access-fmxv5\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:34:09.953739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:09.953735 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab391ca-993d-40cf-bf77-b5ed2618951d-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:34:10.582507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:10.582464 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" event={"ID":"8ab391ca-993d-40cf-bf77-b5ed2618951d","Type":"ContainerDied","Data":"d08109a7f3feba0396e190f4f85dd6d7fcf612bbcfd4b14b8c094c347b715ef6"} Nov 03 22:34:10.582682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:10.582508 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc" Nov 03 22:34:10.582682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:10.582560 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d08109a7f3feba0396e190f4f85dd6d7fcf612bbcfd4b14b8c094c347b715ef6" Nov 03 22:34:12.079816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:12.079787 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:12.463803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:12.463765 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:13.050714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:13.050674 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l"] Nov 03 22:34:13.100785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:13.100748 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8"] Nov 03 22:34:13.106147 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:13.106126 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9ce596-lvp6l"] Nov 03 22:34:13.111333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:13.111311 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8489bb-s57n8"] Nov 03 22:34:13.146967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:13.146923 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0234c2e8-ecd4-4b8e-a523-4cab2501ad3c" path="/var/lib/kubelet/pods/0234c2e8-ecd4-4b8e-a523-4cab2501ad3c/volumes" Nov 03 22:34:13.147527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:13.147503 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044db9d5-efe9-4592-b923-4c86c8203a2a" path="/var/lib/kubelet/pods/044db9d5-efe9-4592-b923-4c86c8203a2a/volumes" Nov 03 22:34:17.029662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:17.029601 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:17.286953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:17.286873 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:17.554809 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:34:17.554729 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:34:21.048761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:21.048724 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc"] Nov 03 22:34:21.099017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:21.098983 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faa7d467-4lfcc"] Nov 03 22:34:21.136172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:21.136130 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04001e8f-c750-4377-be93-7f865cc614fd" path="/var/lib/kubelet/pods/04001e8f-c750-4377-be93-7f865cc614fd/volumes" Nov 03 22:34:21.671931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:21.671897 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:23.023324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:23.023288 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:24.878518 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:24.878487 4298 scope.go:122] "RemoveContainer" containerID="2b9401eaa50a6aa93415e1cd47d72e29b3d4bf54e646be7d0aa0cc0ccd23feba" Nov 03 22:34:24.890155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:24.890134 4298 scope.go:122] "RemoveContainer" containerID="53aa39b8164012354b25d79c0f9c60ad8831ffb89a48f24523023d03bff931f3" Nov 03 22:34:24.900227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:24.900202 4298 scope.go:122] "RemoveContainer" containerID="9a72c0f802eb01577342fde8a20783ae8f66465341b0f9c8e989909e77f5422d" Nov 03 22:34:25.791889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:25.791854 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:26.050022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:26.049941 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m"] Nov 03 22:34:26.100155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:26.100121 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec23852e-mwv2m"] Nov 03 22:34:26.182435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:26.182394 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:27.136028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:27.135990 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2123055a-b7be-4c6d-b422-a44ad5460f42" path="/var/lib/kubelet/pods/2123055a-b7be-4c6d-b422-a44ad5460f42/volumes" Nov 03 22:34:30.678609 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:30.678566 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:32.048990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:32.048954 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q"] Nov 03 22:34:32.099211 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:32.099174 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d23cb25-fd94q"] Nov 03 22:34:33.136333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:33.136288 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288a6772-58f1-4966-ac60-20f47af51fc1" path="/var/lib/kubelet/pods/288a6772-58f1-4966-ac60-20f47af51fc1/volumes" Nov 03 22:34:35.495040 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:35.495004 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:40.485822 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:40.485777 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:44.050272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:44.050235 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5"] Nov 03 22:34:44.100393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:44.100353 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1308704b-rjdt5"] Nov 03 22:34:45.136320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:45.136280 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd349022-5e19-4264-85d7-c8bb5244a0b2" path="/var/lib/kubelet/pods/bd349022-5e19-4264-85d7-c8bb5244a0b2/volumes" Nov 03 22:34:45.177034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:45.177000 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:47.049630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:47.049598 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc"] Nov 03 22:34:47.103657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:47.103608 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc67462e-vklrc"] Nov 03 22:34:47.140053 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:47.140022 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5aad511-b51a-4e53-9259-1865e2d67b61" path="/var/lib/kubelet/pods/e5aad511-b51a-4e53-9259-1865e2d67b61/volumes" Nov 03 22:34:47.553016 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:34:47.552979 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:34:50.110627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:50.110590 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:53.058164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.058130 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42"] Nov 03 22:34:53.058782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.058768 4298 cpu_manager.go:414] "containerMap: removing container" podUID="8ab391ca-993d-40cf-bf77-b5ed2618951d" containerName="revoke-keys" Nov 03 22:34:53.058782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.058783 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab391ca-993d-40cf-bf77-b5ed2618951d" containerName="revoke-keys" Nov 03 22:34:53.058878 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.058811 4298 cpu_manager.go:414] "containerMap: removing container" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" Nov 03 22:34:53.058878 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.058816 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221e35d0-71d1-4488-8108-6dcb28e6b621" containerName="istio-proxy" Nov 03 22:34:53.058950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.058921 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab391ca-993d-40cf-bf77-b5ed2618951d" containerName="revoke-keys" Nov 03 22:34:53.063910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.063891 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:53.117304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.117260 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42"] Nov 03 22:34:53.135724 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.135690 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a083af4-142c-48e3-8476-ec957adca304-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c211881d-jdw42\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:53.135928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.135744 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a083af4-142c-48e3-8476-ec957adca304-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c211881d-jdw42\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:53.135986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.135959 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmddp\" (UniqueName: \"kubernetes.io/projected/1a083af4-142c-48e3-8476-ec957adca304-kube-api-access-gmddp\") pod \"maas-api-revoke-keys-e2e-ait-c211881d-jdw42\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:53.158793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.158760 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:34:53.158957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.158851 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:34:53.237282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.237238 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a083af4-142c-48e3-8476-ec957adca304-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c211881d-jdw42\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:53.237492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.237302 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a083af4-142c-48e3-8476-ec957adca304-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c211881d-jdw42\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:53.237492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.237412 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmddp\" (UniqueName: \"kubernetes.io/projected/1a083af4-142c-48e3-8476-ec957adca304-kube-api-access-gmddp\") pod \"maas-api-revoke-keys-e2e-ait-c211881d-jdw42\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:53.238065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.238040 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a083af4-142c-48e3-8476-ec957adca304-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c211881d-jdw42\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:53.239708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.239676 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a083af4-142c-48e3-8476-ec957adca304-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c211881d-jdw42\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:53.294438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.294403 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmddp\" (UniqueName: \"kubernetes.io/projected/1a083af4-142c-48e3-8476-ec957adca304-kube-api-access-gmddp\") pod \"maas-api-revoke-keys-e2e-ait-c211881d-jdw42\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:53.378747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.378625 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:53.756765 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:34:53.756720 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a083af4_142c_48e3_8476_ec957adca304.slice/crio-469a903427eef1f50a168a76cac2f3fd2a55b91989f16b988646d81be87b9be6 WatchSource:0}: Error finding container 469a903427eef1f50a168a76cac2f3fd2a55b91989f16b988646d81be87b9be6: Status 404 returned error can't find the container with id 469a903427eef1f50a168a76cac2f3fd2a55b91989f16b988646d81be87b9be6 Nov 03 22:34:53.764431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.764399 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42"] Nov 03 22:34:53.975126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:53.975083 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" event={"ID":"1a083af4-142c-48e3-8476-ec957adca304","Type":"ContainerStarted","Data":"469a903427eef1f50a168a76cac2f3fd2a55b91989f16b988646d81be87b9be6"} Nov 03 22:34:54.985296 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:54.985259 4298 generic.go:356] "Generic (PLEG): container finished" podID="1a083af4-142c-48e3-8476-ec957adca304" containerID="939e3bde482500ca1ca6328b8d207179d2cd54c1964e59126c8f2e3550b07642" exitCode=0 Nov 03 22:34:54.985784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:54.985342 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" event={"ID":"1a083af4-142c-48e3-8476-ec957adca304","Type":"ContainerDied","Data":"939e3bde482500ca1ca6328b8d207179d2cd54c1964e59126c8f2e3550b07642"} Nov 03 22:34:55.378285 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:55.378207 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:56.051937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.051903 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn"] Nov 03 22:34:56.105813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.105780 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd"] Nov 03 22:34:56.114634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.114609 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f806c996-996nd"] Nov 03 22:34:56.123366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.123339 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbd6b9-khjnn"] Nov 03 22:34:56.166066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.166039 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:56.284022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.283947 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a083af4-142c-48e3-8476-ec957adca304-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a083af4-142c-48e3-8476-ec957adca304-maas-api-openshift-ca\") pod \"1a083af4-142c-48e3-8476-ec957adca304\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " Nov 03 22:34:56.284192 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.284047 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a083af4-142c-48e3-8476-ec957adca304-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a083af4-142c-48e3-8476-ec957adca304-maas-api-xks-ca\") pod \"1a083af4-142c-48e3-8476-ec957adca304\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " Nov 03 22:34:56.284192 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.284121 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a083af4-142c-48e3-8476-ec957adca304-kube-api-access-gmddp\" (UniqueName: \"kubernetes.io/projected/1a083af4-142c-48e3-8476-ec957adca304-kube-api-access-gmddp\") pod \"1a083af4-142c-48e3-8476-ec957adca304\" (UID: \"1a083af4-142c-48e3-8476-ec957adca304\") " Nov 03 22:34:56.284315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.284289 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a083af4-142c-48e3-8476-ec957adca304-maas-api-openshift-ca" pod "1a083af4-142c-48e3-8476-ec957adca304" (UID: "1a083af4-142c-48e3-8476-ec957adca304"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:34:56.285036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.285009 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a083af4-142c-48e3-8476-ec957adca304-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:34:56.286414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.286381 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a083af4-142c-48e3-8476-ec957adca304-kube-api-access-gmddp" pod "1a083af4-142c-48e3-8476-ec957adca304" (UID: "1a083af4-142c-48e3-8476-ec957adca304"). InnerVolumeSpecName "kube-api-access-gmddp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:56.286534 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.286486 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a083af4-142c-48e3-8476-ec957adca304-maas-api-xks-ca" pod "1a083af4-142c-48e3-8476-ec957adca304" (UID: "1a083af4-142c-48e3-8476-ec957adca304"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:34:56.386684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.386630 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a083af4-142c-48e3-8476-ec957adca304-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:34:56.386684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:56.386685 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmddp\" (UniqueName: \"kubernetes.io/projected/1a083af4-142c-48e3-8476-ec957adca304-kube-api-access-gmddp\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:34:57.001717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:57.001685 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" Nov 03 22:34:57.001717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:57.001705 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42" event={"ID":"1a083af4-142c-48e3-8476-ec957adca304","Type":"ContainerDied","Data":"469a903427eef1f50a168a76cac2f3fd2a55b91989f16b988646d81be87b9be6"} Nov 03 22:34:57.001931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:57.001777 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="469a903427eef1f50a168a76cac2f3fd2a55b91989f16b988646d81be87b9be6" Nov 03 22:34:57.136938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:57.136905 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157b49b4-5df7-421e-9b9b-5318bdba773e" path="/var/lib/kubelet/pods/157b49b4-5df7-421e-9b9b-5318bdba773e/volumes" Nov 03 22:34:57.137418 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:57.137404 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9" path="/var/lib/kubelet/pods/ef819e5c-60a5-48f2-b5e9-ea8ca06aa8a9/volumes" Nov 03 22:34:59.632252 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.632205 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn"] Nov 03 22:34:59.633090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.633067 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1a083af4-142c-48e3-8476-ec957adca304" containerName="revoke-keys" Nov 03 22:34:59.633192 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.633091 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a083af4-142c-48e3-8476-ec957adca304" containerName="revoke-keys" Nov 03 22:34:59.633313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.633297 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a083af4-142c-48e3-8476-ec957adca304" containerName="revoke-keys" Nov 03 22:34:59.638269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.638245 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.690918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.690887 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kf4rv\"" Nov 03 22:34:59.742584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.742545 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.742779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.742621 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.742779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.742701 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.742901 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.742884 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.742940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.742916 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.742985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.742944 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d1ef14-ddb0-43bd-9315-3993276fd02f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.743056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.743032 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.743106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.743068 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-892rc\" (UniqueName: \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-kube-api-access-892rc\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.743166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.743141 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.751198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.751166 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn"] Nov 03 22:34:59.844351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844319 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.844537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844398 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.844537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844438 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.844537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844473 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.844735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844592 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.844735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844610 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.844735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844626 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d1ef14-ddb0-43bd-9315-3993276fd02f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.844735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844712 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.844955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844729 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.844955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844745 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-892rc\" (UniqueName: \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-kube-api-access-892rc\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.844955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844874 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.844955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.844940 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.845276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.845256 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.845422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.845402 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d1ef14-ddb0-43bd-9315-3993276fd02f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.846693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.846675 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.847033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.847016 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.903399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.903310 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.903399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.903384 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-892rc\" (UniqueName: \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-kube-api-access-892rc\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-mfcpn\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:34:59.952065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:34:59.952027 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:35:00.164773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:00.164684 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:00.379203 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:00.379170 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn"] Nov 03 22:35:00.412070 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:35:00.412040 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d1ef14_ddb0_43bd_9315_3993276fd02f.slice/crio-72342a20d5121f8dcf01032e649819c95fde0ed6159d96b3640310f7f05cd4d3 WatchSource:0}: Error finding container 72342a20d5121f8dcf01032e649819c95fde0ed6159d96b3640310f7f05cd4d3: Status 404 returned error can't find the container with id 72342a20d5121f8dcf01032e649819c95fde0ed6159d96b3640310f7f05cd4d3 Nov 03 22:35:00.414853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:00.414795 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:35:00.414933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:00.414860 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:35:00.414933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:00.414891 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:35:01.055957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:01.055911 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" event={"ID":"62d1ef14-ddb0-43bd-9315-3993276fd02f","Type":"ContainerStarted","Data":"d861d071e83a034c8d02b43f77684a502d311450c50ee064c117774d8ca1a751"} Nov 03 22:35:01.056461 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:01.055996 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" event={"ID":"62d1ef14-ddb0-43bd-9315-3993276fd02f","Type":"ContainerStarted","Data":"72342a20d5121f8dcf01032e649819c95fde0ed6159d96b3640310f7f05cd4d3"} Nov 03 22:35:01.058044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:01.058019 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w"] Nov 03 22:35:01.063417 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:01.063386 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a73cd797-bf76w"] Nov 03 22:35:01.157651 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:01.157606 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba540bf-4c1f-455c-a819-6b3e80c0d987" path="/var/lib/kubelet/pods/0ba540bf-4c1f-455c-a819-6b3e80c0d987/volumes" Nov 03 22:35:01.177567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:01.177516 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" podStartSLOduration=2.177501983 podStartE2EDuration="2.177501983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:01.17226995 +0000 UTC m=+1658.153003603" watchObservedRunningTime="2026-11-03 22:35:01.177501983 +0000 UTC m=+1658.158235636" Nov 03 22:35:01.952500 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:01.952460 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:35:01.953722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:01.953698 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:01.953829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:01.953755 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 22:35:02.953112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:02.953025 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:02.953112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:02.953082 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 22:35:03.404865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.404767 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc"] Nov 03 22:35:03.409058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.409029 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:03.463944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.463900 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc"] Nov 03 22:35:03.506126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.506076 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:35:03.506126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.506095 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:35:03.520426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.520380 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ed2f404-24jqc\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:03.520607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.520572 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ed2f404-24jqc\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:03.520734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.520705 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh8l4\" (UniqueName: \"kubernetes.io/projected/931311a4-60e0-4d4c-a60d-0714824c1933-kube-api-access-vh8l4\") pod \"maas-api-revoke-sub-4ed2f404-24jqc\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:03.622443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.622393 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ed2f404-24jqc\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:03.622625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.622513 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ed2f404-24jqc\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:03.622625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.622555 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh8l4\" (UniqueName: \"kubernetes.io/projected/931311a4-60e0-4d4c-a60d-0714824c1933-kube-api-access-vh8l4\") pod \"maas-api-revoke-sub-4ed2f404-24jqc\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:03.623287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.623259 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ed2f404-24jqc\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:03.649231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.649192 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ed2f404-24jqc\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:03.679839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.679766 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh8l4\" (UniqueName: \"kubernetes.io/projected/931311a4-60e0-4d4c-a60d-0714824c1933-kube-api-access-vh8l4\") pod \"maas-api-revoke-sub-4ed2f404-24jqc\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:03.746668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.735875 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:03.868664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.866484 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:03.968441 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.966426 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:03.968441 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:03.966487 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 22:35:04.157059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:04.157031 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc"] Nov 03 22:35:04.656715 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:04.649765 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:04.866691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:04.866612 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:04.954130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:04.954084 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:04.954458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:04.954425 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 22:35:05.092553 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:05.092516 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" event={"ID":"931311a4-60e0-4d4c-a60d-0714824c1933","Type":"ContainerStarted","Data":"cd333ba3b66bb6ab0cf92b72df8bed700be3a3895bff4984485d395b23d5afa1"} Nov 03 22:35:05.094127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:05.094090 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" event={"ID":"931311a4-60e0-4d4c-a60d-0714824c1933","Type":"ContainerStarted","Data":"c10546f742a36e6787cc9370a171b9a9e50d1ed7ee13c624abff5726ed65eaec"} Nov 03 22:35:05.153671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:05.152747 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:05.208766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:05.208627 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" podStartSLOduration=2.208609389 podStartE2EDuration="2.208609389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:05.207421669 +0000 UTC m=+1662.188155370" watchObservedRunningTime="2026-11-03 22:35:05.208609389 +0000 UTC m=+1662.189343063" Nov 03 22:35:05.955166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:05.955125 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:05.955459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:05.955431 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 22:35:05.996745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:05.996484 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:06.102046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:06.102013 4298 generic.go:356] "Generic (PLEG): container finished" podID="931311a4-60e0-4d4c-a60d-0714824c1933" containerID="cd333ba3b66bb6ab0cf92b72df8bed700be3a3895bff4984485d395b23d5afa1" exitCode=0 Nov 03 22:35:06.102473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:06.102104 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" event={"ID":"931311a4-60e0-4d4c-a60d-0714824c1933","Type":"ContainerDied","Data":"cd333ba3b66bb6ab0cf92b72df8bed700be3a3895bff4984485d395b23d5afa1"} Nov 03 22:35:06.712978 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:06.707756 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:06.952787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:06.952745 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:06.952997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:06.952816 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 22:35:07.401577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.401489 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:07.495238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.493680 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-openshift-ca\") pod \"931311a4-60e0-4d4c-a60d-0714824c1933\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " Nov 03 22:35:07.495238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.493928 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931311a4-60e0-4d4c-a60d-0714824c1933-kube-api-access-vh8l4\" (UniqueName: \"kubernetes.io/projected/931311a4-60e0-4d4c-a60d-0714824c1933-kube-api-access-vh8l4\") pod \"931311a4-60e0-4d4c-a60d-0714824c1933\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " Nov 03 22:35:07.495238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.494080 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-xks-ca\") pod \"931311a4-60e0-4d4c-a60d-0714824c1933\" (UID: \"931311a4-60e0-4d4c-a60d-0714824c1933\") " Nov 03 22:35:07.498023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.495912 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-openshift-ca" pod "931311a4-60e0-4d4c-a60d-0714824c1933" (UID: "931311a4-60e0-4d4c-a60d-0714824c1933"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:07.499140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.499069 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-xks-ca" pod "931311a4-60e0-4d4c-a60d-0714824c1933" (UID: "931311a4-60e0-4d4c-a60d-0714824c1933"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:35:07.500033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.499996 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931311a4-60e0-4d4c-a60d-0714824c1933-kube-api-access-vh8l4" pod "931311a4-60e0-4d4c-a60d-0714824c1933" (UID: "931311a4-60e0-4d4c-a60d-0714824c1933"). InnerVolumeSpecName "kube-api-access-vh8l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:07.511898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.511864 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:07.596567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.596521 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:07.596567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.596568 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931311a4-60e0-4d4c-a60d-0714824c1933-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:07.596885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.596582 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh8l4\" (UniqueName: \"kubernetes.io/projected/931311a4-60e0-4d4c-a60d-0714824c1933-kube-api-access-vh8l4\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:07.953215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.953179 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:07.953385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:07.953244 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 22:35:08.116170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:08.116134 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:08.136702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:08.136672 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" Nov 03 22:35:08.136884 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:08.136672 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc" event={"ID":"931311a4-60e0-4d4c-a60d-0714824c1933","Type":"ContainerDied","Data":"c10546f742a36e6787cc9370a171b9a9e50d1ed7ee13c624abff5726ed65eaec"} Nov 03 22:35:08.136884 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:08.136821 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c10546f742a36e6787cc9370a171b9a9e50d1ed7ee13c624abff5726ed65eaec" Nov 03 22:35:08.822150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:08.822112 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7"] Nov 03 22:35:08.822695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:08.822403 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" podUID="12c643fd-8534-4e35-808f-6d40aaa2e04c" containerName="istio-proxy" containerID="cri-o://b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969" gracePeriod=30 Nov 03 22:35:09.098719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:09.098565 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:09.781446 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:09.781412 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:35:09.782015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:09.781980 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:35:09.782846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:09.782826 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:35:10.609360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:10.609321 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:11.363462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:11.363427 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:11.555917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:11.555882 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:12.473153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:12.473122 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:14.092403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.092306 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:14.102219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.102191 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:35:14.119282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119252 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-token\" (UniqueName: \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-token\") pod \"12c643fd-8534-4e35-808f-6d40aaa2e04c\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " Nov 03 22:35:14.119438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119292 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-kube-api-access-6pdnw\" (UniqueName: \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-kube-api-access-6pdnw\") pod \"12c643fd-8534-4e35-808f-6d40aaa2e04c\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " Nov 03 22:35:14.119438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119322 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c643fd-8534-4e35-808f-6d40aaa2e04c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c643fd-8534-4e35-808f-6d40aaa2e04c-istiod-ca-cert\") pod \"12c643fd-8534-4e35-808f-6d40aaa2e04c\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " Nov 03 22:35:14.119438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119414 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-podinfo\") pod \"12c643fd-8534-4e35-808f-6d40aaa2e04c\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " Nov 03 22:35:14.119553 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119463 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-certs\") pod \"12c643fd-8534-4e35-808f-6d40aaa2e04c\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " Nov 03 22:35:14.119553 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119522 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-socket\") pod \"12c643fd-8534-4e35-808f-6d40aaa2e04c\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " Nov 03 22:35:14.119651 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119606 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-credential-socket\") pod \"12c643fd-8534-4e35-808f-6d40aaa2e04c\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " Nov 03 22:35:14.120013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119789 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-envoy\") pod \"12c643fd-8534-4e35-808f-6d40aaa2e04c\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " Nov 03 22:35:14.120013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119775 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-certs" pod "12c643fd-8534-4e35-808f-6d40aaa2e04c" (UID: "12c643fd-8534-4e35-808f-6d40aaa2e04c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.120013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119856 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-socket" pod "12c643fd-8534-4e35-808f-6d40aaa2e04c" (UID: "12c643fd-8534-4e35-808f-6d40aaa2e04c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.120013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119872 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-data\") pod \"12c643fd-8534-4e35-808f-6d40aaa2e04c\" (UID: \"12c643fd-8534-4e35-808f-6d40aaa2e04c\") " Nov 03 22:35:14.120013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.119942 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c643fd-8534-4e35-808f-6d40aaa2e04c-istiod-ca-cert" pod "12c643fd-8534-4e35-808f-6d40aaa2e04c" (UID: "12c643fd-8534-4e35-808f-6d40aaa2e04c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:14.120251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.120052 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-credential-socket" pod "12c643fd-8534-4e35-808f-6d40aaa2e04c" (UID: "12c643fd-8534-4e35-808f-6d40aaa2e04c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.120251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.120095 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-data" pod "12c643fd-8534-4e35-808f-6d40aaa2e04c" (UID: "12c643fd-8534-4e35-808f-6d40aaa2e04c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.120655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.120623 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c643fd-8534-4e35-808f-6d40aaa2e04c-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.120698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.120665 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.120698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.120682 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.120770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.120696 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.120770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.120709 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.121727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.121702 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-podinfo" pod "12c643fd-8534-4e35-808f-6d40aaa2e04c" (UID: "12c643fd-8534-4e35-808f-6d40aaa2e04c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:14.121817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.121721 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-token" pod "12c643fd-8534-4e35-808f-6d40aaa2e04c" (UID: "12c643fd-8534-4e35-808f-6d40aaa2e04c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:14.122005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.121983 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-kube-api-access-6pdnw" pod "12c643fd-8534-4e35-808f-6d40aaa2e04c" (UID: "12c643fd-8534-4e35-808f-6d40aaa2e04c"). InnerVolumeSpecName "kube-api-access-6pdnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:14.122103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.122086 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-envoy" pod "12c643fd-8534-4e35-808f-6d40aaa2e04c" (UID: "12c643fd-8534-4e35-808f-6d40aaa2e04c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.192116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.192078 4298 generic.go:356] "Generic (PLEG): container finished" podID="12c643fd-8534-4e35-808f-6d40aaa2e04c" containerID="b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969" exitCode=0 Nov 03 22:35:14.192261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.192132 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" Nov 03 22:35:14.192261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.192161 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" event={"ID":"12c643fd-8534-4e35-808f-6d40aaa2e04c","Type":"ContainerDied","Data":"b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969"} Nov 03 22:35:14.192261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.192220 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7" event={"ID":"12c643fd-8534-4e35-808f-6d40aaa2e04c","Type":"ContainerDied","Data":"5227f7a40b85289fe5b1e11f21980f5ff61868e81e9b537d3c88157b6d84dbfc"} Nov 03 22:35:14.192261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.192247 4298 scope.go:122] "RemoveContainer" containerID="b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969" Nov 03 22:35:14.204097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.204074 4298 scope.go:122] "RemoveContainer" containerID="b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969" Nov 03 22:35:14.204401 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:35:14.204380 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969\": container with ID starting with b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969 not found: ID does not exist" containerID="b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969" Nov 03 22:35:14.204443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.204410 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969"} err="failed to get container status \"b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969\": rpc error: code = NotFound desc = could not find container \"b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969\": container with ID starting with b9fabc8ce71c44e6de1d95b1042c3577f3fb6d940e40d469afebb77f23b90969 not found: ID does not exist" Nov 03 22:35:14.222111 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.222085 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.222111 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.222108 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.222310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.222118 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pdnw\" (UniqueName: \"kubernetes.io/projected/12c643fd-8534-4e35-808f-6d40aaa2e04c-kube-api-access-6pdnw\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.222310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.222128 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c643fd-8534-4e35-808f-6d40aaa2e04c-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.389796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.389696 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7"] Nov 03 22:35:14.395540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.395511 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-rjvl7"] Nov 03 22:35:14.768929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.768894 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn"] Nov 03 22:35:14.769584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.769564 4298 cpu_manager.go:414] "containerMap: removing container" podUID="931311a4-60e0-4d4c-a60d-0714824c1933" containerName="revoke-keys" Nov 03 22:35:14.769740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.769584 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931311a4-60e0-4d4c-a60d-0714824c1933" containerName="revoke-keys" Nov 03 22:35:14.769740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.769631 4298 cpu_manager.go:414] "containerMap: removing container" podUID="12c643fd-8534-4e35-808f-6d40aaa2e04c" containerName="istio-proxy" Nov 03 22:35:14.769740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.769659 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c643fd-8534-4e35-808f-6d40aaa2e04c" containerName="istio-proxy" Nov 03 22:35:14.769902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.769823 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c643fd-8534-4e35-808f-6d40aaa2e04c" containerName="istio-proxy" Nov 03 22:35:14.769902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.769848 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="931311a4-60e0-4d4c-a60d-0714824c1933" containerName="revoke-keys" Nov 03 22:35:14.774740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.774721 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.841660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.837910 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6jn\" (UniqueName: \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-kube-api-access-hb6jn\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.841660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.838051 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.841660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.838103 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.841660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.838155 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.841660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.838191 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.841660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.838245 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.841660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.838276 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.841660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.838309 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.841660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.838355 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.841660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.840163 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn"] Nov 03 22:35:14.939765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.939730 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.939907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.939783 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.939907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.939828 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.939907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.939885 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.940030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.939950 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6jn\" (UniqueName: \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-kube-api-access-hb6jn\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.940068 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.940047 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.940121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.940099 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.940235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.940175 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.940235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.940220 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.940357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.940237 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.940591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.940568 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.940668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.940600 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.940668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.940632 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.940762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.940745 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.942525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.942502 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.942611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.942594 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.996984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.996951 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:14.997122 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:14.997032 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6jn\" (UniqueName: \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-kube-api-access-hb6jn\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-td7bn\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:15.088841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:15.088754 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:15.137750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:15.137717 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c643fd-8534-4e35-808f-6d40aaa2e04c" path="/var/lib/kubelet/pods/12c643fd-8534-4e35-808f-6d40aaa2e04c/volumes" Nov 03 22:35:15.531980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:15.531944 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:35:15.532100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:15.532020 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:35:15.532100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:15.532047 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:35:15.536667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:15.536624 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn"] Nov 03 22:35:15.586969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:15.586938 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:16.214181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:16.214137 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" event={"ID":"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db","Type":"ContainerStarted","Data":"53a7b6639c0265b9a7fb6324b98269cf367f205d4948e3ab7ca3bae5658ce072"} Nov 03 22:35:16.214588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:16.214199 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" event={"ID":"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db","Type":"ContainerStarted","Data":"b0a464a33b9108e30de1acddd1d5b3c0f46db159f042b3eb2f206553d6a2e505"} Nov 03 22:35:16.449842 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:16.449800 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:16.658812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:16.648268 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:17.089455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:17.089416 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:17.090695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:17.090656 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-td7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:35:17.090844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:17.090720 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 22:35:17.706094 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:35:17.702215 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:35:18.090173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:18.090088 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-td7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:35:18.090173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:18.090154 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 22:35:19.089257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:19.089221 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-td7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:35:19.089627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:19.089279 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 22:35:20.089417 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:20.089379 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-td7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:35:20.089829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:20.089435 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 22:35:20.352805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:20.352280 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:20.607097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:20.607026 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:21.089390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:21.089354 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-td7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:35:21.089568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:21.089405 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 22:35:22.090029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:22.089992 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-td7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:35:22.090519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:22.090079 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 22:35:23.089542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:23.089510 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-td7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:35:23.089741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:23.089560 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 22:35:24.245487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:24.245442 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:24.246024 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:24.245943 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:24.246690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:24.246670 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:35:24.414135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:24.414082 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" podStartSLOduration=10.414066564 podStartE2EDuration="10.414066564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:16.448938017 +0000 UTC m=+1673.429671670" watchObservedRunningTime="2026-11-03 22:35:24.414066564 +0000 UTC m=+1681.394800216" Nov 03 22:35:24.956318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:24.956271 4298 scope.go:122] "RemoveContainer" containerID="4abe882594d9cd339a20ed561aeb712e180cb921d7f425be2131788eaa4abfdb" Nov 03 22:35:24.966783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:24.966759 4298 scope.go:122] "RemoveContainer" containerID="6d734eca6d88d51bf563d4c0d704a20f05545d6452fac4c18b9ab52f01c92b25" Nov 03 22:35:24.977783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:24.977759 4298 scope.go:122] "RemoveContainer" containerID="0d456ceebfa2f0f20e87815d808c1aed54635258d6b13a551fb409bbc1b02772" Nov 03 22:35:24.988140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:24.988115 4298 scope.go:122] "RemoveContainer" containerID="4c25d6e0a139f4fc251501243556c9e4b85a77a35a2b3290119aa52b71d397dc" Nov 03 22:35:24.998106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:24.998084 4298 scope.go:122] "RemoveContainer" containerID="eb0c251bad35e933d5585619e3b0acc1231a0b01c782156fb8e085b63ea396d5" Nov 03 22:35:25.007886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:25.007866 4298 scope.go:122] "RemoveContainer" containerID="cb8400922a0555c4c552707a00a3402f7b06d84bcd209d94ce273e81405d38f3" Nov 03 22:35:25.017431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:25.017404 4298 scope.go:122] "RemoveContainer" containerID="45c4751860a810dc6a55ec205d7ec38f5f23da1603680a7c8f29d5cc37c13216" Nov 03 22:35:25.026580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:25.026559 4298 scope.go:122] "RemoveContainer" containerID="5064c4b231a91e48980dd8c760ed4f14e239d8bf0d1528f9f62923611d5df3ff" Nov 03 22:35:25.055450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:25.055422 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m"] Nov 03 22:35:25.061472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:25.061452 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3218de71-rlj2m"] Nov 03 22:35:25.073110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:25.073080 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:25.140026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:25.139990 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81cf9ba0-c45b-4fba-92ba-23fba424c30f" path="/var/lib/kubelet/pods/81cf9ba0-c45b-4fba-92ba-23fba424c30f/volumes" Nov 03 22:35:25.902345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:25.902308 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w"] Nov 03 22:35:25.902872 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:25.902500 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" containerID="cri-o://8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7" gracePeriod=30 Nov 03 22:35:29.565357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:29.565324 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:31.057509 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.057457 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4"] Nov 03 22:35:31.105445 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.105410 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ebc0ff3-g9wl4"] Nov 03 22:35:31.138432 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.138378 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163fc7a1-7ca2-46ef-9313-208fec197592" path="/var/lib/kubelet/pods/163fc7a1-7ca2-46ef-9313-208fec197592/volumes" Nov 03 22:35:31.202370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.202345 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:35:31.353908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.353855 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-socket\") pod \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " Nov 03 22:35:31.354114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.353938 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-token\" (UniqueName: \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-token\") pod \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " Nov 03 22:35:31.354114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.353960 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-certs\") pod \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " Nov 03 22:35:31.354114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.353997 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-kube-api-access-fbjwm\" (UniqueName: \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-kube-api-access-fbjwm\") pod \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " Nov 03 22:35:31.354114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.354041 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-credential-socket\") pod \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " Nov 03 22:35:31.354114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.354080 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-podinfo\") pod \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " Nov 03 22:35:31.354414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.354197 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-envoy\") pod \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " Nov 03 22:35:31.354414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.354191 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-socket" pod "d03ed135-ca08-446d-b9ee-f36ece2b49c8" (UID: "d03ed135-ca08-446d-b9ee-f36ece2b49c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:31.354414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.354234 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-data\") pod \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " Nov 03 22:35:31.354414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.354257 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istiod-ca-cert\") pod \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\" (UID: \"d03ed135-ca08-446d-b9ee-f36ece2b49c8\") " Nov 03 22:35:31.354894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.354860 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-certs" pod "d03ed135-ca08-446d-b9ee-f36ece2b49c8" (UID: "d03ed135-ca08-446d-b9ee-f36ece2b49c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:31.354894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.354884 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-data" pod "d03ed135-ca08-446d-b9ee-f36ece2b49c8" (UID: "d03ed135-ca08-446d-b9ee-f36ece2b49c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:31.355140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.355112 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-credential-socket" pod "d03ed135-ca08-446d-b9ee-f36ece2b49c8" (UID: "d03ed135-ca08-446d-b9ee-f36ece2b49c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:31.355208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.355171 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.355208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.355194 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.355319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.355211 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.355319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.355208 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istiod-ca-cert" pod "d03ed135-ca08-446d-b9ee-f36ece2b49c8" (UID: "d03ed135-ca08-446d-b9ee-f36ece2b49c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:31.356596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.356568 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-podinfo" pod "d03ed135-ca08-446d-b9ee-f36ece2b49c8" (UID: "d03ed135-ca08-446d-b9ee-f36ece2b49c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:31.356881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.356852 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-kube-api-access-fbjwm" pod "d03ed135-ca08-446d-b9ee-f36ece2b49c8" (UID: "d03ed135-ca08-446d-b9ee-f36ece2b49c8"). InnerVolumeSpecName "kube-api-access-fbjwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:31.357276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.357255 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-envoy" pod "d03ed135-ca08-446d-b9ee-f36ece2b49c8" (UID: "d03ed135-ca08-446d-b9ee-f36ece2b49c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:31.357506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.357485 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-token" pod "d03ed135-ca08-446d-b9ee-f36ece2b49c8" (UID: "d03ed135-ca08-446d-b9ee-f36ece2b49c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:31.370301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.370271 4298 generic.go:356] "Generic (PLEG): container finished" podID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerID="8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7" exitCode=0 Nov 03 22:35:31.370439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.370337 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" Nov 03 22:35:31.370439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.370357 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" event={"ID":"d03ed135-ca08-446d-b9ee-f36ece2b49c8","Type":"ContainerDied","Data":"8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7"} Nov 03 22:35:31.370439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.370409 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w" event={"ID":"d03ed135-ca08-446d-b9ee-f36ece2b49c8","Type":"ContainerDied","Data":"d12da4da4cc1edeebe7b2b1daf9fb09554b80da708cced150e1d1517843416cc"} Nov 03 22:35:31.370439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.370436 4298 scope.go:122] "RemoveContainer" containerID="8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7" Nov 03 22:35:31.390688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.390619 4298 scope.go:122] "RemoveContainer" containerID="8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7" Nov 03 22:35:31.391014 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:35:31.390987 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7\": container with ID starting with 8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7 not found: ID does not exist" containerID="8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7" Nov 03 22:35:31.391111 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.391024 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7"} err="failed to get container status \"8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7\": rpc error: code = NotFound desc = could not find container \"8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7\": container with ID starting with 8074e9ec3b62c67d5f735132af2d3e6d3801132032c2eadf4f7842fded1562e7 not found: ID does not exist" Nov 03 22:35:31.456916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.456887 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.456916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.456913 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbjwm\" (UniqueName: \"kubernetes.io/projected/d03ed135-ca08-446d-b9ee-f36ece2b49c8-kube-api-access-fbjwm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.457140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.456925 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.457140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.456934 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.457140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.456943 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.457140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.456951 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03ed135-ca08-446d-b9ee-f36ece2b49c8-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.540875 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.540844 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w"] Nov 03 22:35:31.546700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:31.546664 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-m274w"] Nov 03 22:35:33.138689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:33.138633 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" path="/var/lib/kubelet/pods/d03ed135-ca08-446d-b9ee-f36ece2b49c8/volumes" Nov 03 22:35:33.514542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:33.514507 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:35.019356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:35.019317 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:35.290703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:35.290574 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:35.483530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:35.483480 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:35.515739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:35.514838 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42"] Nov 03 22:35:35.563566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:35.563484 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c211881d-jdw42"] Nov 03 22:35:36.084753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:36.084726 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:37.145212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:37.145169 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a083af4-142c-48e3-8476-ec957adca304" path="/var/lib/kubelet/pods/1a083af4-142c-48e3-8476-ec957adca304/volumes" Nov 03 22:35:40.243241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:40.243209 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:40.794513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:40.794481 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:41.064620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:41.064537 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r"] Nov 03 22:35:41.111741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:41.111708 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6781acfd-jt82r"] Nov 03 22:35:41.145577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:41.145545 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589c4892-f1f4-4930-b6c0-867f9dbaa5d7" path="/var/lib/kubelet/pods/589c4892-f1f4-4930-b6c0-867f9dbaa5d7/volumes" Nov 03 22:35:41.948389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:41.948346 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:42.146872 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:42.146837 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:44.780888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:44.780813 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:46.626365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:46.626321 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:46.815058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:46.815025 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:47.553951 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:35:47.553913 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:35:47.903428 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:47.903356 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq"] Nov 03 22:35:47.904044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:47.904029 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" Nov 03 22:35:47.904098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:47.904046 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" Nov 03 22:35:47.904173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:47.904163 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03ed135-ca08-446d-b9ee-f36ece2b49c8" containerName="istio-proxy" Nov 03 22:35:47.907686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:47.907670 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:47.965192 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:47.965162 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq"] Nov 03 22:35:48.009191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.009160 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:35:48.009336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.009224 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:35:48.024346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.024317 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mpv\" (UniqueName: \"kubernetes.io/projected/8582c1e0-3305-4331-9a89-6aa46a71ae1b-kube-api-access-45mpv\") pod \"maas-api-revoke-sub-b90a2563-pk7nq\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:48.024516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.024359 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b90a2563-pk7nq\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:48.024516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.024470 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b90a2563-pk7nq\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:48.126037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.126004 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mpv\" (UniqueName: \"kubernetes.io/projected/8582c1e0-3305-4331-9a89-6aa46a71ae1b-kube-api-access-45mpv\") pod \"maas-api-revoke-sub-b90a2563-pk7nq\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:48.126188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.126045 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b90a2563-pk7nq\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:48.126188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.126120 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b90a2563-pk7nq\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:48.126800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.126776 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b90a2563-pk7nq\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:48.128485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.128463 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b90a2563-pk7nq\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:48.182951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.182882 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mpv\" (UniqueName: \"kubernetes.io/projected/8582c1e0-3305-4331-9a89-6aa46a71ae1b-kube-api-access-45mpv\") pod \"maas-api-revoke-sub-b90a2563-pk7nq\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:48.221865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.221836 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:48.591696 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:35:48.591661 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8582c1e0_3305_4331_9a89_6aa46a71ae1b.slice/crio-70f384f79ac2d21a650a20fb03f80e9ed9543761d5655bf3bcf6bdf229d121c7 WatchSource:0}: Error finding container 70f384f79ac2d21a650a20fb03f80e9ed9543761d5655bf3bcf6bdf229d121c7: Status 404 returned error can't find the container with id 70f384f79ac2d21a650a20fb03f80e9ed9543761d5655bf3bcf6bdf229d121c7 Nov 03 22:35:48.596618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:48.596595 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq"] Nov 03 22:35:49.548540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:49.548498 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" event={"ID":"8582c1e0-3305-4331-9a89-6aa46a71ae1b","Type":"ContainerStarted","Data":"9a50e343457ce979f165d041284c9109d0eb3b47be9a16d0b61775f584a2b25e"} Nov 03 22:35:49.549022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:49.548594 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" event={"ID":"8582c1e0-3305-4331-9a89-6aa46a71ae1b","Type":"ContainerStarted","Data":"70f384f79ac2d21a650a20fb03f80e9ed9543761d5655bf3bcf6bdf229d121c7"} Nov 03 22:35:49.589582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:49.589548 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:49.666951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:49.666893 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" podStartSLOduration=2.666878784 podStartE2EDuration="2.666878784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:49.661587356 +0000 UTC m=+1706.642321012" watchObservedRunningTime="2026-11-03 22:35:49.666878784 +0000 UTC m=+1706.647612436" Nov 03 22:35:49.998797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:49.998761 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:50.667225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:50.667150 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:50.675814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:50.675782 4298 generic.go:356] "Generic (PLEG): container finished" podID="8582c1e0-3305-4331-9a89-6aa46a71ae1b" containerID="9a50e343457ce979f165d041284c9109d0eb3b47be9a16d0b61775f584a2b25e" exitCode=0 Nov 03 22:35:50.676429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:50.676402 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" event={"ID":"8582c1e0-3305-4331-9a89-6aa46a71ae1b","Type":"ContainerDied","Data":"9a50e343457ce979f165d041284c9109d0eb3b47be9a16d0b61775f584a2b25e"} Nov 03 22:35:51.317498 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:51.317464 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:51.504962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:51.504932 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:52.016766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.016718 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:52.152256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.152220 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-openshift-ca\") pod \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " Nov 03 22:35:52.152467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.152346 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-xks-ca\") pod \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " Nov 03 22:35:52.152467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.152395 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8582c1e0-3305-4331-9a89-6aa46a71ae1b-kube-api-access-45mpv\" (UniqueName: \"kubernetes.io/projected/8582c1e0-3305-4331-9a89-6aa46a71ae1b-kube-api-access-45mpv\") pod \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\" (UID: \"8582c1e0-3305-4331-9a89-6aa46a71ae1b\") " Nov 03 22:35:52.152699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.152625 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-openshift-ca" pod "8582c1e0-3305-4331-9a89-6aa46a71ae1b" (UID: "8582c1e0-3305-4331-9a89-6aa46a71ae1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:52.153147 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.153123 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:52.155858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.155827 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-xks-ca" pod "8582c1e0-3305-4331-9a89-6aa46a71ae1b" (UID: "8582c1e0-3305-4331-9a89-6aa46a71ae1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:35:52.163717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.163681 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8582c1e0-3305-4331-9a89-6aa46a71ae1b-kube-api-access-45mpv" pod "8582c1e0-3305-4331-9a89-6aa46a71ae1b" (UID: "8582c1e0-3305-4331-9a89-6aa46a71ae1b"). InnerVolumeSpecName "kube-api-access-45mpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:52.254990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.254900 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8582c1e0-3305-4331-9a89-6aa46a71ae1b-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:52.254990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.254935 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45mpv\" (UniqueName: \"kubernetes.io/projected/8582c1e0-3305-4331-9a89-6aa46a71ae1b-kube-api-access-45mpv\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:52.698050 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.698019 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" Nov 03 22:35:52.698050 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.698040 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq" event={"ID":"8582c1e0-3305-4331-9a89-6aa46a71ae1b","Type":"ContainerDied","Data":"70f384f79ac2d21a650a20fb03f80e9ed9543761d5655bf3bcf6bdf229d121c7"} Nov 03 22:35:52.698260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:52.698089 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70f384f79ac2d21a650a20fb03f80e9ed9543761d5655bf3bcf6bdf229d121c7" Nov 03 22:35:54.447981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:54.447950 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:54.617139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:54.617102 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:54.824091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:54.823996 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:55.108167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:55.108071 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn"] Nov 03 22:35:55.108362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:55.108337 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" containerID="cri-o://de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7" gracePeriod=30 Nov 03 22:35:55.156973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:55.156936 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp"] Nov 03 22:35:55.157175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:55.157151 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" containerID="cri-o://81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b" gracePeriod=30 Nov 03 22:35:55.162562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:55.162535 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6"] Nov 03 22:35:55.162830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:55.162796 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerName="istio-proxy" containerID="cri-o://a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7" gracePeriod=30 Nov 03 22:35:55.206940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:55.206910 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj"] Nov 03 22:35:55.207122 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:55.207104 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerName="istio-proxy" containerID="cri-o://634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd" gracePeriod=30 Nov 03 22:35:56.390777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:56.390739 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt"] Nov 03 22:35:56.391271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:56.391004 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" containerID="cri-o://f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8" gracePeriod=30 Nov 03 22:35:56.448413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:56.448372 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t"] Nov 03 22:35:56.459400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:56.459360 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx"] Nov 03 22:35:56.459784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:56.459706 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" containerName="istio-proxy" containerID="cri-o://3945917e0bb4418c951fa3a3dc4f9b80ac4801067e203b8f73086f42e146faf9" gracePeriod=30 Nov 03 22:35:56.463287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:56.463262 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:56.495404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:56.495373 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k"] Nov 03 22:35:56.495676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:56.495613 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" containerID="cri-o://ce6967efc6f1800252af16607d97b4e8f0a86e6359eb4dcd6b7d91c58515be3c" gracePeriod=30 Nov 03 22:35:56.663976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:56.663892 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:56.730233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:56.730195 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" containerName="istio-proxy" containerID="cri-o://a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f" gracePeriod=30 Nov 03 22:35:56.855316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:56.855282 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:58.697067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:58.697029 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:59.300048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:59.300015 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:35:59.300215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:59.300076 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:35:59.374051 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:59.374011 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:35:59.374224 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:59.374102 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:35:59.492808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:35:59.492774 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:00.265554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.265527 4298 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:36:00.265890 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.265608 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:36:00.451942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.451917 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:36:00.489878 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.489848 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-podinfo\") pod \"d8c87f27-faa4-4ce3-8c24-939548b06801\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " Nov 03 22:36:00.490033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.489922 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-envoy\") pod \"d8c87f27-faa4-4ce3-8c24-939548b06801\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " Nov 03 22:36:00.490033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.489942 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-kube-api-access-7b58s\" (UniqueName: \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-kube-api-access-7b58s\") pod \"d8c87f27-faa4-4ce3-8c24-939548b06801\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " Nov 03 22:36:00.490033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.489992 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-token\" (UniqueName: \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-token\") pod \"d8c87f27-faa4-4ce3-8c24-939548b06801\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " Nov 03 22:36:00.490033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.490017 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c87f27-faa4-4ce3-8c24-939548b06801-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c87f27-faa4-4ce3-8c24-939548b06801-istiod-ca-cert\") pod \"d8c87f27-faa4-4ce3-8c24-939548b06801\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " Nov 03 22:36:00.490259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.490044 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-certs\") pod \"d8c87f27-faa4-4ce3-8c24-939548b06801\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " Nov 03 22:36:00.490259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.490113 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-data\") pod \"d8c87f27-faa4-4ce3-8c24-939548b06801\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " Nov 03 22:36:00.490259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.490215 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-socket\") pod \"d8c87f27-faa4-4ce3-8c24-939548b06801\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " Nov 03 22:36:00.490259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.490234 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-credential-socket\") pod \"d8c87f27-faa4-4ce3-8c24-939548b06801\" (UID: \"d8c87f27-faa4-4ce3-8c24-939548b06801\") " Nov 03 22:36:00.491024 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.490990 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-credential-socket" pod "d8c87f27-faa4-4ce3-8c24-939548b06801" (UID: "d8c87f27-faa4-4ce3-8c24-939548b06801"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.491587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.491560 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c87f27-faa4-4ce3-8c24-939548b06801-istiod-ca-cert" pod "d8c87f27-faa4-4ce3-8c24-939548b06801" (UID: "d8c87f27-faa4-4ce3-8c24-939548b06801"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:00.494242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.494200 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-podinfo" pod "d8c87f27-faa4-4ce3-8c24-939548b06801" (UID: "d8c87f27-faa4-4ce3-8c24-939548b06801"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:00.494423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.494394 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-socket" pod "d8c87f27-faa4-4ce3-8c24-939548b06801" (UID: "d8c87f27-faa4-4ce3-8c24-939548b06801"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.494523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.494457 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-envoy" pod "d8c87f27-faa4-4ce3-8c24-939548b06801" (UID: "d8c87f27-faa4-4ce3-8c24-939548b06801"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.494523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.494472 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-data" pod "d8c87f27-faa4-4ce3-8c24-939548b06801" (UID: "d8c87f27-faa4-4ce3-8c24-939548b06801"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.494607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.494523 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-certs" pod "d8c87f27-faa4-4ce3-8c24-939548b06801" (UID: "d8c87f27-faa4-4ce3-8c24-939548b06801"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.495059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.494942 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-kube-api-access-7b58s" pod "d8c87f27-faa4-4ce3-8c24-939548b06801" (UID: "d8c87f27-faa4-4ce3-8c24-939548b06801"). InnerVolumeSpecName "kube-api-access-7b58s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:00.495152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.495124 4298 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:36:00.495201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.495168 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:36:00.498468 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.498443 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-token" pod "d8c87f27-faa4-4ce3-8c24-939548b06801" (UID: "d8c87f27-faa4-4ce3-8c24-939548b06801"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:00.548996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.548962 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:36:00.591800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.591771 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-socket\") pod \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " Nov 03 22:36:00.591949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.591816 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-envoy\") pod \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " Nov 03 22:36:00.591949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.591856 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-credential-socket\") pod \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " Nov 03 22:36:00.591949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.591925 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-certs\") pod \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " Nov 03 22:36:00.592096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.592066 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-podinfo\") pod \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " Nov 03 22:36:00.592142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.592105 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istiod-ca-cert\") pod \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " Nov 03 22:36:00.592276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.592249 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-credential-socket" pod "992ba108-76e1-45ee-b87b-7fe5dcd799b5" (UID: "992ba108-76e1-45ee-b87b-7fe5dcd799b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.592346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.592293 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-token\" (UniqueName: \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-token\") pod \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " Nov 03 22:36:00.592346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.592301 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-certs" pod "992ba108-76e1-45ee-b87b-7fe5dcd799b5" (UID: "992ba108-76e1-45ee-b87b-7fe5dcd799b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.592346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.592325 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-data\") pod \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " Nov 03 22:36:00.592520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.592437 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-kube-api-access-292tk\" (UniqueName: \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-kube-api-access-292tk\") pod \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\" (UID: \"992ba108-76e1-45ee-b87b-7fe5dcd799b5\") " Nov 03 22:36:00.592720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.592623 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istiod-ca-cert" pod "992ba108-76e1-45ee-b87b-7fe5dcd799b5" (UID: "992ba108-76e1-45ee-b87b-7fe5dcd799b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:00.592827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.592806 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-data" pod "992ba108-76e1-45ee-b87b-7fe5dcd799b5" (UID: "992ba108-76e1-45ee-b87b-7fe5dcd799b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.593107 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593086 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593111 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593127 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593142 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593154 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593167 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593181 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593195 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593208 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b58s\" (UniqueName: \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-kube-api-access-7b58s\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593222 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c87f27-faa4-4ce3-8c24-939548b06801-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593235 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c87f27-faa4-4ce3-8c24-939548b06801-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593254 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c87f27-faa4-4ce3-8c24-939548b06801-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593267 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.593885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.593583 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-socket" pod "992ba108-76e1-45ee-b87b-7fe5dcd799b5" (UID: "992ba108-76e1-45ee-b87b-7fe5dcd799b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.594536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.594512 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-podinfo" pod "992ba108-76e1-45ee-b87b-7fe5dcd799b5" (UID: "992ba108-76e1-45ee-b87b-7fe5dcd799b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:00.595119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.595092 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-token" pod "992ba108-76e1-45ee-b87b-7fe5dcd799b5" (UID: "992ba108-76e1-45ee-b87b-7fe5dcd799b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:00.595248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.595233 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-kube-api-access-292tk" pod "992ba108-76e1-45ee-b87b-7fe5dcd799b5" (UID: "992ba108-76e1-45ee-b87b-7fe5dcd799b5"). InnerVolumeSpecName "kube-api-access-292tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:00.595351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.595337 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-envoy" pod "992ba108-76e1-45ee-b87b-7fe5dcd799b5" (UID: "992ba108-76e1-45ee-b87b-7fe5dcd799b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.601322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.601300 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:36:00.603517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.603502 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:36:00.693928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.693840 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-token\" (UniqueName: \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-token\") pod \"ac669eb9-9135-44df-858a-7fd3c7238e28\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " Nov 03 22:36:00.693928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.693889 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac669eb9-9135-44df-858a-7fd3c7238e28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac669eb9-9135-44df-858a-7fd3c7238e28-istiod-ca-cert\") pod \"ac669eb9-9135-44df-858a-7fd3c7238e28\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " Nov 03 22:36:00.693928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.693919 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-token\" (UniqueName: \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-token\") pod \"4d2411c9-7385-4b74-b850-2006e89a8f54\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " Nov 03 22:36:00.694237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.693948 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-certs\") pod \"4d2411c9-7385-4b74-b850-2006e89a8f54\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " Nov 03 22:36:00.694237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.693979 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-kube-api-access-bsz9q\" (UniqueName: \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-kube-api-access-bsz9q\") pod \"ac669eb9-9135-44df-858a-7fd3c7238e28\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " Nov 03 22:36:00.694237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694076 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-envoy\") pod \"ac669eb9-9135-44df-858a-7fd3c7238e28\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " Nov 03 22:36:00.694237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694099 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-certs\") pod \"ac669eb9-9135-44df-858a-7fd3c7238e28\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " Nov 03 22:36:00.694237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694124 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-data\") pod \"4d2411c9-7385-4b74-b850-2006e89a8f54\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " Nov 03 22:36:00.694237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694231 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d2411c9-7385-4b74-b850-2006e89a8f54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2411c9-7385-4b74-b850-2006e89a8f54-istiod-ca-cert\") pod \"4d2411c9-7385-4b74-b850-2006e89a8f54\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " Nov 03 22:36:00.694560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694286 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-podinfo\") pod \"ac669eb9-9135-44df-858a-7fd3c7238e28\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " Nov 03 22:36:00.694560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694336 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-credential-socket\") pod \"ac669eb9-9135-44df-858a-7fd3c7238e28\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " Nov 03 22:36:00.694560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694364 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-kube-api-access-znsmv\" (UniqueName: \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-kube-api-access-znsmv\") pod \"4d2411c9-7385-4b74-b850-2006e89a8f54\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " Nov 03 22:36:00.694560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694409 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-socket\") pod \"4d2411c9-7385-4b74-b850-2006e89a8f54\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " Nov 03 22:36:00.694560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694255 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-certs" pod "4d2411c9-7385-4b74-b850-2006e89a8f54" (UID: "4d2411c9-7385-4b74-b850-2006e89a8f54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.694560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694478 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-data" pod "4d2411c9-7385-4b74-b850-2006e89a8f54" (UID: "4d2411c9-7385-4b74-b850-2006e89a8f54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.694560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694465 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-certs" pod "ac669eb9-9135-44df-858a-7fd3c7238e28" (UID: "ac669eb9-9135-44df-858a-7fd3c7238e28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.694560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694498 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-credential-socket\") pod \"4d2411c9-7385-4b74-b850-2006e89a8f54\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " Nov 03 22:36:00.694560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694415 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac669eb9-9135-44df-858a-7fd3c7238e28-istiod-ca-cert" pod "ac669eb9-9135-44df-858a-7fd3c7238e28" (UID: "ac669eb9-9135-44df-858a-7fd3c7238e28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:00.694970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694596 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-podinfo\") pod \"4d2411c9-7385-4b74-b850-2006e89a8f54\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " Nov 03 22:36:00.694970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694627 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-socket\") pod \"ac669eb9-9135-44df-858a-7fd3c7238e28\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " Nov 03 22:36:00.694970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694735 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-data\") pod \"ac669eb9-9135-44df-858a-7fd3c7238e28\" (UID: \"ac669eb9-9135-44df-858a-7fd3c7238e28\") " Nov 03 22:36:00.694970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694764 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-envoy\") pod \"4d2411c9-7385-4b74-b850-2006e89a8f54\" (UID: \"4d2411c9-7385-4b74-b850-2006e89a8f54\") " Nov 03 22:36:00.694970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.694955 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2411c9-7385-4b74-b850-2006e89a8f54-istiod-ca-cert" pod "4d2411c9-7385-4b74-b850-2006e89a8f54" (UID: "4d2411c9-7385-4b74-b850-2006e89a8f54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:00.695471 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695429 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-credential-socket" pod "ac669eb9-9135-44df-858a-7fd3c7238e28" (UID: "ac669eb9-9135-44df-858a-7fd3c7238e28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.695628 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695608 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.695735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695632 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.695735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695664 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2411c9-7385-4b74-b850-2006e89a8f54-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.695735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695687 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.695735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695701 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.695735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695713 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.696074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695728 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac669eb9-9135-44df-858a-7fd3c7238e28-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.696074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695758 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.696074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695778 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-292tk\" (UniqueName: \"kubernetes.io/projected/992ba108-76e1-45ee-b87b-7fe5dcd799b5-kube-api-access-292tk\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.696074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695793 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.696074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.695808 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992ba108-76e1-45ee-b87b-7fe5dcd799b5-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.696741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.696714 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-token" pod "4d2411c9-7385-4b74-b850-2006e89a8f54" (UID: "4d2411c9-7385-4b74-b850-2006e89a8f54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:00.697431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.696953 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-socket" pod "4d2411c9-7385-4b74-b850-2006e89a8f54" (UID: "4d2411c9-7385-4b74-b850-2006e89a8f54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.697431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.697079 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-token" pod "ac669eb9-9135-44df-858a-7fd3c7238e28" (UID: "ac669eb9-9135-44df-858a-7fd3c7238e28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:00.697431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.697163 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-credential-socket" pod "4d2411c9-7385-4b74-b850-2006e89a8f54" (UID: "4d2411c9-7385-4b74-b850-2006e89a8f54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.697431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.697372 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-socket" pod "ac669eb9-9135-44df-858a-7fd3c7238e28" (UID: "ac669eb9-9135-44df-858a-7fd3c7238e28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.697431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.697389 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-data" pod "ac669eb9-9135-44df-858a-7fd3c7238e28" (UID: "ac669eb9-9135-44df-858a-7fd3c7238e28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.697856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.697695 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-envoy" pod "4d2411c9-7385-4b74-b850-2006e89a8f54" (UID: "4d2411c9-7385-4b74-b850-2006e89a8f54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.697856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.697740 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-kube-api-access-bsz9q" pod "ac669eb9-9135-44df-858a-7fd3c7238e28" (UID: "ac669eb9-9135-44df-858a-7fd3c7238e28"). InnerVolumeSpecName "kube-api-access-bsz9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:00.698302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.698282 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-kube-api-access-znsmv" pod "4d2411c9-7385-4b74-b850-2006e89a8f54" (UID: "4d2411c9-7385-4b74-b850-2006e89a8f54"). InnerVolumeSpecName "kube-api-access-znsmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:00.698469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.698455 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-podinfo" pod "4d2411c9-7385-4b74-b850-2006e89a8f54" (UID: "4d2411c9-7385-4b74-b850-2006e89a8f54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:00.698729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.698710 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-podinfo" pod "ac669eb9-9135-44df-858a-7fd3c7238e28" (UID: "ac669eb9-9135-44df-858a-7fd3c7238e28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:00.698996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.698982 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-envoy" pod "ac669eb9-9135-44df-858a-7fd3c7238e28" (UID: "ac669eb9-9135-44df-858a-7fd3c7238e28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:00.766321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.766288 4298 generic.go:356] "Generic (PLEG): container finished" podID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerID="81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b" exitCode=0 Nov 03 22:36:00.766537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.766387 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" event={"ID":"4d2411c9-7385-4b74-b850-2006e89a8f54","Type":"ContainerDied","Data":"81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b"} Nov 03 22:36:00.766537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.766410 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" Nov 03 22:36:00.766537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.766468 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp" event={"ID":"4d2411c9-7385-4b74-b850-2006e89a8f54","Type":"ContainerDied","Data":"b19fded1198468c22ab7b5e5493cfed5551db896a612d28b399908bf2d3ed8b8"} Nov 03 22:36:00.766745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.766543 4298 scope.go:122] "RemoveContainer" containerID="81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b" Nov 03 22:36:00.768008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.767985 4298 generic.go:356] "Generic (PLEG): container finished" podID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerID="a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7" exitCode=0 Nov 03 22:36:00.768092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.768032 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" Nov 03 22:36:00.768092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.768067 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" event={"ID":"992ba108-76e1-45ee-b87b-7fe5dcd799b5","Type":"ContainerDied","Data":"a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7"} Nov 03 22:36:00.768231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.768116 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6" event={"ID":"992ba108-76e1-45ee-b87b-7fe5dcd799b5","Type":"ContainerDied","Data":"8dede78b6cdf17c3f475c365b73fd97f5699612bd25f264e5aa69f18ce80b7c6"} Nov 03 22:36:00.771175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.771150 4298 generic.go:356] "Generic (PLEG): container finished" podID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerID="634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd" exitCode=0 Nov 03 22:36:00.771285 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.771184 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" event={"ID":"ac669eb9-9135-44df-858a-7fd3c7238e28","Type":"ContainerDied","Data":"634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd"} Nov 03 22:36:00.771285 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.771201 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" Nov 03 22:36:00.771285 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.771236 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj" event={"ID":"ac669eb9-9135-44df-858a-7fd3c7238e28","Type":"ContainerDied","Data":"8900ddfe153355fd388ac723b4ede9a0b9d6f402523f2ad98e84d6ab7809b38a"} Nov 03 22:36:00.776041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.776014 4298 generic.go:356] "Generic (PLEG): container finished" podID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerID="de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7" exitCode=0 Nov 03 22:36:00.776143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.776061 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" event={"ID":"d8c87f27-faa4-4ce3-8c24-939548b06801","Type":"ContainerDied","Data":"de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7"} Nov 03 22:36:00.776143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.776091 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" Nov 03 22:36:00.776143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.776097 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn" event={"ID":"d8c87f27-faa4-4ce3-8c24-939548b06801","Type":"ContainerDied","Data":"76fa50a592fbb46132b4dca8be195b83cd7518f08f311adbd4cfac62d6e33828"} Nov 03 22:36:00.779410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.779335 4298 scope.go:122] "RemoveContainer" containerID="81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b" Nov 03 22:36:00.779679 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:00.779613 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b\": container with ID starting with 81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b not found: ID does not exist" containerID="81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b" Nov 03 22:36:00.779769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.779672 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b"} err="failed to get container status \"81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b\": rpc error: code = NotFound desc = could not find container \"81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b\": container with ID starting with 81dfb82ca2e7a05fbe6418adda80851c8a166f19b0094ec2808704fb17278d4b not found: ID does not exist" Nov 03 22:36:00.779769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.779695 4298 scope.go:122] "RemoveContainer" containerID="a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7" Nov 03 22:36:00.798334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798305 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.798483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798340 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.798483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798364 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znsmv\" (UniqueName: \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-kube-api-access-znsmv\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.798483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798380 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.798483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798395 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.798483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798408 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.798483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798427 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.798483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798441 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.798483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798466 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.798483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798479 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.799205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798497 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2411c9-7385-4b74-b850-2006e89a8f54-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.799205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.798511 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsz9q\" (UniqueName: \"kubernetes.io/projected/ac669eb9-9135-44df-858a-7fd3c7238e28-kube-api-access-bsz9q\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:00.812954 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.812905 4298 scope.go:122] "RemoveContainer" containerID="a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7" Nov 03 22:36:00.813522 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:00.813477 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7\": container with ID starting with a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7 not found: ID does not exist" containerID="a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7" Nov 03 22:36:00.813636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.813549 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7"} err="failed to get container status \"a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7\": rpc error: code = NotFound desc = could not find container \"a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7\": container with ID starting with a1c8388d21a5eccb9f2f694ee4d3f5e3755644368ddeed840810a4e2d9f11cb7 not found: ID does not exist" Nov 03 22:36:00.813636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.813588 4298 scope.go:122] "RemoveContainer" containerID="634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd" Nov 03 22:36:00.825362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.825281 4298 scope.go:122] "RemoveContainer" containerID="634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd" Nov 03 22:36:00.825703 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:00.825628 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd\": container with ID starting with 634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd not found: ID does not exist" containerID="634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd" Nov 03 22:36:00.825788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.825715 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd"} err="failed to get container status \"634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd\": rpc error: code = NotFound desc = could not find container \"634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd\": container with ID starting with 634288c758842f8e34965fbe76a3a4336d9640e0ee840c7b44716bdf9bfbf3dd not found: ID does not exist" Nov 03 22:36:00.825788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.825741 4298 scope.go:122] "RemoveContainer" containerID="de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7" Nov 03 22:36:00.836458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.836433 4298 scope.go:122] "RemoveContainer" containerID="de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7" Nov 03 22:36:00.836803 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:00.836779 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7\": container with ID starting with de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7 not found: ID does not exist" containerID="de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7" Nov 03 22:36:00.836894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:00.836809 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7"} err="failed to get container status \"de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7\": rpc error: code = NotFound desc = could not find container \"de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7\": container with ID starting with de4ad472e1bc029dcd9bd8379c138a92dec6f6359edb94878140ff3251ddf9c7 not found: ID does not exist" Nov 03 22:36:01.143305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.143277 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6"] Nov 03 22:36:01.148032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.148008 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-pm2z6"] Nov 03 22:36:01.252134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.252100 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp"] Nov 03 22:36:01.257030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.257004 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-2r2xp"] Nov 03 22:36:01.364332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.364299 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj"] Nov 03 22:36:01.369337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.369309 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-kblvj"] Nov 03 22:36:01.478597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.478561 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn"] Nov 03 22:36:01.483474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.483448 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-x5wfn"] Nov 03 22:36:01.771313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.771275 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:36:01.787175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.786955 4298 generic.go:356] "Generic (PLEG): container finished" podID="914760e1-f094-48ab-848f-e0e168fb265e" containerID="ce6967efc6f1800252af16607d97b4e8f0a86e6359eb4dcd6b7d91c58515be3c" exitCode=0 Nov 03 22:36:01.787175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.787099 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" event={"ID":"914760e1-f094-48ab-848f-e0e168fb265e","Type":"ContainerDied","Data":"ce6967efc6f1800252af16607d97b4e8f0a86e6359eb4dcd6b7d91c58515be3c"} Nov 03 22:36:01.789005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.788976 4298 generic.go:356] "Generic (PLEG): container finished" podID="debadcac-80e7-4843-a84a-dc60f364566c" containerID="f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8" exitCode=0 Nov 03 22:36:01.789164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.789085 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" event={"ID":"debadcac-80e7-4843-a84a-dc60f364566c","Type":"ContainerDied","Data":"f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8"} Nov 03 22:36:01.789164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.789133 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" event={"ID":"debadcac-80e7-4843-a84a-dc60f364566c","Type":"ContainerDied","Data":"3f14b1c4fa78cfa9c707c53f7bce18165f759d8e95c84b92de8d82b1388c289e"} Nov 03 22:36:01.789164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.789162 4298 scope.go:122] "RemoveContainer" containerID="f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8" Nov 03 22:36:01.789337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.789286 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt" Nov 03 22:36:01.792572 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.792430 4298 generic.go:356] "Generic (PLEG): container finished" podID="e55d3001-b0d1-41a6-a376-8f057710801b" containerID="3945917e0bb4418c951fa3a3dc4f9b80ac4801067e203b8f73086f42e146faf9" exitCode=0 Nov 03 22:36:01.792572 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.792475 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" event={"ID":"e55d3001-b0d1-41a6-a376-8f057710801b","Type":"ContainerDied","Data":"3945917e0bb4418c951fa3a3dc4f9b80ac4801067e203b8f73086f42e146faf9"} Nov 03 22:36:01.815311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.815270 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/debadcac-80e7-4843-a84a-dc60f364566c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debadcac-80e7-4843-a84a-dc60f364566c-istio-podinfo\") pod \"debadcac-80e7-4843-a84a-dc60f364566c\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " Nov 03 22:36:01.815481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.815350 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-istio-token\" (UniqueName: \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-istio-token\") pod \"debadcac-80e7-4843-a84a-dc60f364566c\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " Nov 03 22:36:01.815481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.815376 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-kube-api-access-lp7sb\" (UniqueName: \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-kube-api-access-lp7sb\") pod \"debadcac-80e7-4843-a84a-dc60f364566c\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " Nov 03 22:36:01.815618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.815526 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-certs\") pod \"debadcac-80e7-4843-a84a-dc60f364566c\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " Nov 03 22:36:01.815618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.815573 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-envoy\") pod \"debadcac-80e7-4843-a84a-dc60f364566c\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " Nov 03 22:36:01.815806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.815779 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-certs" pod "debadcac-80e7-4843-a84a-dc60f364566c" (UID: "debadcac-80e7-4843-a84a-dc60f364566c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.815880 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.815813 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-socket\") pod \"debadcac-80e7-4843-a84a-dc60f364566c\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " Nov 03 22:36:01.815880 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.815855 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-credential-socket\") pod \"debadcac-80e7-4843-a84a-dc60f364566c\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " Nov 03 22:36:01.815975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.815915 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-data\") pod \"debadcac-80e7-4843-a84a-dc60f364566c\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " Nov 03 22:36:01.815975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.815961 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/debadcac-80e7-4843-a84a-dc60f364566c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debadcac-80e7-4843-a84a-dc60f364566c-istiod-ca-cert\") pod \"debadcac-80e7-4843-a84a-dc60f364566c\" (UID: \"debadcac-80e7-4843-a84a-dc60f364566c\") " Nov 03 22:36:01.816079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.816047 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-socket" pod "debadcac-80e7-4843-a84a-dc60f364566c" (UID: "debadcac-80e7-4843-a84a-dc60f364566c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.816201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.816149 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-credential-socket" pod "debadcac-80e7-4843-a84a-dc60f364566c" (UID: "debadcac-80e7-4843-a84a-dc60f364566c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.816262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.816208 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-data" pod "debadcac-80e7-4843-a84a-dc60f364566c" (UID: "debadcac-80e7-4843-a84a-dc60f364566c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.816412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.816387 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debadcac-80e7-4843-a84a-dc60f364566c-istiod-ca-cert" pod "debadcac-80e7-4843-a84a-dc60f364566c" (UID: "debadcac-80e7-4843-a84a-dc60f364566c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:01.816734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.816715 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.816820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.816738 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.816820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.816752 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.816820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.816766 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.816820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.816780 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debadcac-80e7-4843-a84a-dc60f364566c-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.818092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.818052 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-envoy" pod "debadcac-80e7-4843-a84a-dc60f364566c" (UID: "debadcac-80e7-4843-a84a-dc60f364566c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.818212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.818177 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-istio-token" pod "debadcac-80e7-4843-a84a-dc60f364566c" (UID: "debadcac-80e7-4843-a84a-dc60f364566c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:01.818330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.818307 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/debadcac-80e7-4843-a84a-dc60f364566c-istio-podinfo" pod "debadcac-80e7-4843-a84a-dc60f364566c" (UID: "debadcac-80e7-4843-a84a-dc60f364566c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:01.818330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.818318 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-kube-api-access-lp7sb" pod "debadcac-80e7-4843-a84a-dc60f364566c" (UID: "debadcac-80e7-4843-a84a-dc60f364566c"). InnerVolumeSpecName "kube-api-access-lp7sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:01.844959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.844939 4298 scope.go:122] "RemoveContainer" containerID="f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8" Nov 03 22:36:01.845211 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:01.845185 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8\": container with ID starting with f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8 not found: ID does not exist" containerID="f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8" Nov 03 22:36:01.845274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.845223 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8"} err="failed to get container status \"f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8\": rpc error: code = NotFound desc = could not find container \"f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8\": container with ID starting with f872b3bfc31f35fe7e067b68f3c4de807770884284bc7d9e4a9a77f637ba27c8 not found: ID does not exist" Nov 03 22:36:01.845512 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.845496 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:36:01.848570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.848550 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:36:01.918194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918159 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914760e1-f094-48ab-848f-e0e168fb265e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914760e1-f094-48ab-848f-e0e168fb265e-istiod-ca-cert\") pod \"914760e1-f094-48ab-848f-e0e168fb265e\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " Nov 03 22:36:01.918390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918207 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-data\") pod \"e55d3001-b0d1-41a6-a376-8f057710801b\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " Nov 03 22:36:01.918390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918242 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-certs\") pod \"914760e1-f094-48ab-848f-e0e168fb265e\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " Nov 03 22:36:01.918390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918293 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55d3001-b0d1-41a6-a376-8f057710801b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e55d3001-b0d1-41a6-a376-8f057710801b-istiod-ca-cert\") pod \"e55d3001-b0d1-41a6-a376-8f057710801b\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " Nov 03 22:36:01.918390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918332 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-credential-socket\") pod \"e55d3001-b0d1-41a6-a376-8f057710801b\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " Nov 03 22:36:01.918390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918376 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-istio-token\" (UniqueName: \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-istio-token\") pod \"e55d3001-b0d1-41a6-a376-8f057710801b\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " Nov 03 22:36:01.918687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918431 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-kube-api-access-g9j76\" (UniqueName: \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-kube-api-access-g9j76\") pod \"e55d3001-b0d1-41a6-a376-8f057710801b\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " Nov 03 22:36:01.918687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918524 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-certs\") pod \"e55d3001-b0d1-41a6-a376-8f057710801b\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " Nov 03 22:36:01.918687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918577 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-istio-token\" (UniqueName: \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-istio-token\") pod \"914760e1-f094-48ab-848f-e0e168fb265e\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " Nov 03 22:36:01.918687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918636 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-envoy\") pod \"914760e1-f094-48ab-848f-e0e168fb265e\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " Nov 03 22:36:01.918909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918690 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-kube-api-access-d948h\" (UniqueName: \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-kube-api-access-d948h\") pod \"914760e1-f094-48ab-848f-e0e168fb265e\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " Nov 03 22:36:01.918909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918731 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55d3001-b0d1-41a6-a376-8f057710801b-istiod-ca-cert" pod "e55d3001-b0d1-41a6-a376-8f057710801b" (UID: "e55d3001-b0d1-41a6-a376-8f057710801b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:01.918909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918741 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-credential-socket" pod "e55d3001-b0d1-41a6-a376-8f057710801b" (UID: "e55d3001-b0d1-41a6-a376-8f057710801b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.918909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918772 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-envoy\") pod \"e55d3001-b0d1-41a6-a376-8f057710801b\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " Nov 03 22:36:01.918909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918841 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-socket\") pod \"914760e1-f094-48ab-848f-e0e168fb265e\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " Nov 03 22:36:01.919163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918918 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-socket\") pod \"e55d3001-b0d1-41a6-a376-8f057710801b\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " Nov 03 22:36:01.919163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918988 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-certs" pod "914760e1-f094-48ab-848f-e0e168fb265e" (UID: "914760e1-f094-48ab-848f-e0e168fb265e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.919163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.918995 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-data\") pod \"914760e1-f094-48ab-848f-e0e168fb265e\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " Nov 03 22:36:01.919163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.919008 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-data" pod "e55d3001-b0d1-41a6-a376-8f057710801b" (UID: "e55d3001-b0d1-41a6-a376-8f057710801b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.919163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.919058 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-credential-socket\") pod \"914760e1-f094-48ab-848f-e0e168fb265e\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " Nov 03 22:36:01.919163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.919101 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/914760e1-f094-48ab-848f-e0e168fb265e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914760e1-f094-48ab-848f-e0e168fb265e-istio-podinfo\") pod \"914760e1-f094-48ab-848f-e0e168fb265e\" (UID: \"914760e1-f094-48ab-848f-e0e168fb265e\") " Nov 03 22:36:01.919469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.919168 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-data" pod "914760e1-f094-48ab-848f-e0e168fb265e" (UID: "914760e1-f094-48ab-848f-e0e168fb265e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.919469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.919181 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e55d3001-b0d1-41a6-a376-8f057710801b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e55d3001-b0d1-41a6-a376-8f057710801b-istio-podinfo\") pod \"e55d3001-b0d1-41a6-a376-8f057710801b\" (UID: \"e55d3001-b0d1-41a6-a376-8f057710801b\") " Nov 03 22:36:01.919580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.919495 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914760e1-f094-48ab-848f-e0e168fb265e-istiod-ca-cert" pod "914760e1-f094-48ab-848f-e0e168fb265e" (UID: "914760e1-f094-48ab-848f-e0e168fb265e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:01.920030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.920004 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debadcac-80e7-4843-a84a-dc60f364566c-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.921659 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.921684 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.921698 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp7sb\" (UniqueName: \"kubernetes.io/projected/debadcac-80e7-4843-a84a-dc60f364566c-kube-api-access-lp7sb\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.921713 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914760e1-f094-48ab-848f-e0e168fb265e-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.921726 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.921741 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.921754 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e55d3001-b0d1-41a6-a376-8f057710801b-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.921769 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.921782 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debadcac-80e7-4843-a84a-dc60f364566c-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.920936 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-credential-socket" pod "914760e1-f094-48ab-848f-e0e168fb265e" (UID: "914760e1-f094-48ab-848f-e0e168fb265e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.921131 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-certs" pod "e55d3001-b0d1-41a6-a376-8f057710801b" (UID: "e55d3001-b0d1-41a6-a376-8f057710801b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.922052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.921399 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-socket" pod "914760e1-f094-48ab-848f-e0e168fb265e" (UID: "914760e1-f094-48ab-848f-e0e168fb265e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.922751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.922633 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-envoy" pod "914760e1-f094-48ab-848f-e0e168fb265e" (UID: "914760e1-f094-48ab-848f-e0e168fb265e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.922910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.922856 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-socket" pod "e55d3001-b0d1-41a6-a376-8f057710801b" (UID: "e55d3001-b0d1-41a6-a376-8f057710801b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.928880 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.928850 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-istio-token" pod "914760e1-f094-48ab-848f-e0e168fb265e" (UID: "914760e1-f094-48ab-848f-e0e168fb265e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:01.930146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.930106 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-kube-api-access-d948h" pod "914760e1-f094-48ab-848f-e0e168fb265e" (UID: "914760e1-f094-48ab-848f-e0e168fb265e"). InnerVolumeSpecName "kube-api-access-d948h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:01.930545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.930519 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/914760e1-f094-48ab-848f-e0e168fb265e-istio-podinfo" pod "914760e1-f094-48ab-848f-e0e168fb265e" (UID: "914760e1-f094-48ab-848f-e0e168fb265e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:01.930625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.930566 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-istio-token" pod "e55d3001-b0d1-41a6-a376-8f057710801b" (UID: "e55d3001-b0d1-41a6-a376-8f057710801b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:01.930625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.930599 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-kube-api-access-g9j76" pod "e55d3001-b0d1-41a6-a376-8f057710801b" (UID: "e55d3001-b0d1-41a6-a376-8f057710801b"). InnerVolumeSpecName "kube-api-access-g9j76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:01.933729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.933693 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-envoy" pod "e55d3001-b0d1-41a6-a376-8f057710801b" (UID: "e55d3001-b0d1-41a6-a376-8f057710801b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:01.933916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:01.933889 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e55d3001-b0d1-41a6-a376-8f057710801b-istio-podinfo" pod "e55d3001-b0d1-41a6-a376-8f057710801b" (UID: "e55d3001-b0d1-41a6-a376-8f057710801b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:02.019825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.019796 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:36:02.023304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023278 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.023389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023307 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9j76\" (UniqueName: \"kubernetes.io/projected/e55d3001-b0d1-41a6-a376-8f057710801b-kube-api-access-g9j76\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.023389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023318 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.023389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023329 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.023389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023338 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.023389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023346 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d948h\" (UniqueName: \"kubernetes.io/projected/914760e1-f094-48ab-848f-e0e168fb265e-kube-api-access-d948h\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.023389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023355 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.023389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023363 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.023389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023373 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55d3001-b0d1-41a6-a376-8f057710801b-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.023389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023383 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914760e1-f094-48ab-848f-e0e168fb265e-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.023389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023391 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914760e1-f094-48ab-848f-e0e168fb265e-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.023732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.023400 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e55d3001-b0d1-41a6-a376-8f057710801b-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.124765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.124732 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-socket\") pod \"9a156fac-b1b4-4cf1-a546-658009fec833\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " Nov 03 22:36:02.124944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.124811 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-envoy\") pod \"9a156fac-b1b4-4cf1-a546-658009fec833\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " Nov 03 22:36:02.124944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.124913 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-kube-api-access-2jp6p\" (UniqueName: \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-kube-api-access-2jp6p\") pod \"9a156fac-b1b4-4cf1-a546-658009fec833\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " Nov 03 22:36:02.125075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.124944 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-certs\") pod \"9a156fac-b1b4-4cf1-a546-658009fec833\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " Nov 03 22:36:02.125075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.124997 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-istio-token\" (UniqueName: \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-istio-token\") pod \"9a156fac-b1b4-4cf1-a546-658009fec833\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " Nov 03 22:36:02.125180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.125067 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-socket" pod "9a156fac-b1b4-4cf1-a546-658009fec833" (UID: "9a156fac-b1b4-4cf1-a546-658009fec833"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:02.125234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.125178 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-credential-socket\") pod \"9a156fac-b1b4-4cf1-a546-658009fec833\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " Nov 03 22:36:02.125289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.125245 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-data\") pod \"9a156fac-b1b4-4cf1-a546-658009fec833\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " Nov 03 22:36:02.125348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.125290 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a156fac-b1b4-4cf1-a546-658009fec833-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a156fac-b1b4-4cf1-a546-658009fec833-istiod-ca-cert\") pod \"9a156fac-b1b4-4cf1-a546-658009fec833\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " Nov 03 22:36:02.125348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.125330 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a156fac-b1b4-4cf1-a546-658009fec833-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a156fac-b1b4-4cf1-a546-658009fec833-istio-podinfo\") pod \"9a156fac-b1b4-4cf1-a546-658009fec833\" (UID: \"9a156fac-b1b4-4cf1-a546-658009fec833\") " Nov 03 22:36:02.126021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.125868 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.126219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.126145 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-certs" pod "9a156fac-b1b4-4cf1-a546-658009fec833" (UID: "9a156fac-b1b4-4cf1-a546-658009fec833"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:02.126445 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.126411 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-data" pod "9a156fac-b1b4-4cf1-a546-658009fec833" (UID: "9a156fac-b1b4-4cf1-a546-658009fec833"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:02.126746 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.126704 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a156fac-b1b4-4cf1-a546-658009fec833-istiod-ca-cert" pod "9a156fac-b1b4-4cf1-a546-658009fec833" (UID: "9a156fac-b1b4-4cf1-a546-658009fec833"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:02.127139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.127114 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-credential-socket" pod "9a156fac-b1b4-4cf1-a546-658009fec833" (UID: "9a156fac-b1b4-4cf1-a546-658009fec833"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:02.128339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.128304 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-kube-api-access-2jp6p" pod "9a156fac-b1b4-4cf1-a546-658009fec833" (UID: "9a156fac-b1b4-4cf1-a546-658009fec833"). InnerVolumeSpecName "kube-api-access-2jp6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:02.128447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.128428 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-envoy" pod "9a156fac-b1b4-4cf1-a546-658009fec833" (UID: "9a156fac-b1b4-4cf1-a546-658009fec833"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:02.128540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.128515 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a156fac-b1b4-4cf1-a546-658009fec833-istio-podinfo" pod "9a156fac-b1b4-4cf1-a546-658009fec833" (UID: "9a156fac-b1b4-4cf1-a546-658009fec833"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:02.129599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.129578 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-istio-token" pod "9a156fac-b1b4-4cf1-a546-658009fec833" (UID: "9a156fac-b1b4-4cf1-a546-658009fec833"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:02.227988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.227795 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a156fac-b1b4-4cf1-a546-658009fec833-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.227988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.227992 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.228253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.228008 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jp6p\" (UniqueName: \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-kube-api-access-2jp6p\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.228253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.228019 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.228253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.228029 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a156fac-b1b4-4cf1-a546-658009fec833-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.228253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.228037 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.228253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.228047 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a156fac-b1b4-4cf1-a546-658009fec833-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.228253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.228055 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a156fac-b1b4-4cf1-a546-658009fec833-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:02.424616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.424537 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt"] Nov 03 22:36:02.429462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.429427 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-624lt"] Nov 03 22:36:02.807164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.807128 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" Nov 03 22:36:02.807164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.807158 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k" event={"ID":"914760e1-f094-48ab-848f-e0e168fb265e","Type":"ContainerDied","Data":"00fe43e61847c17a8f9368798ddc62728ef759332a1d2538cb2624f63354a344"} Nov 03 22:36:02.807383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.807227 4298 scope.go:122] "RemoveContainer" containerID="ce6967efc6f1800252af16607d97b4e8f0a86e6359eb4dcd6b7d91c58515be3c" Nov 03 22:36:02.809947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.809920 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" Nov 03 22:36:02.810066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.809938 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx" event={"ID":"e55d3001-b0d1-41a6-a376-8f057710801b","Type":"ContainerDied","Data":"760cc8603bf2f9c8f77cd770f70a39b0b2c60cfb43299128f966d6e6c2833746"} Nov 03 22:36:02.811522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.811499 4298 generic.go:356] "Generic (PLEG): container finished" podID="9a156fac-b1b4-4cf1-a546-658009fec833" containerID="a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f" exitCode=0 Nov 03 22:36:02.811659 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.811552 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" Nov 03 22:36:02.811659 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.811580 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" event={"ID":"9a156fac-b1b4-4cf1-a546-658009fec833","Type":"ContainerDied","Data":"a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f"} Nov 03 22:36:02.811812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.811675 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t" event={"ID":"9a156fac-b1b4-4cf1-a546-658009fec833","Type":"ContainerDied","Data":"823d964a58e2ae9c55a436f24912e8b6d0ad190773ce96e67771af763a8e3e6e"} Nov 03 22:36:02.820496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.820478 4298 scope.go:122] "RemoveContainer" containerID="3945917e0bb4418c951fa3a3dc4f9b80ac4801067e203b8f73086f42e146faf9" Nov 03 22:36:02.835424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.835402 4298 scope.go:122] "RemoveContainer" containerID="a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f" Nov 03 22:36:02.846216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.846197 4298 scope.go:122] "RemoveContainer" containerID="a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f" Nov 03 22:36:02.846585 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:02.846554 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f\": container with ID starting with a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f not found: ID does not exist" containerID="a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f" Nov 03 22:36:02.846781 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.846592 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f"} err="failed to get container status \"a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f\": rpc error: code = NotFound desc = could not find container \"a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f\": container with ID starting with a7bd7e705197d88dda8bcf37dbb35d8beac74dc09a38791446d0dce3e50b804f not found: ID does not exist" Nov 03 22:36:02.984750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.984689 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx"] Nov 03 22:36:02.986323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:02.986293 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-kqvgx"] Nov 03 22:36:03.089180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.089095 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k"] Nov 03 22:36:03.093662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.093626 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-fjp7k"] Nov 03 22:36:03.136341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.136308 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" path="/var/lib/kubelet/pods/4d2411c9-7385-4b74-b850-2006e89a8f54/volumes" Nov 03 22:36:03.136977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.136960 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914760e1-f094-48ab-848f-e0e168fb265e" path="/var/lib/kubelet/pods/914760e1-f094-48ab-848f-e0e168fb265e/volumes" Nov 03 22:36:03.137396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.137383 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" path="/var/lib/kubelet/pods/992ba108-76e1-45ee-b87b-7fe5dcd799b5/volumes" Nov 03 22:36:03.138135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.138117 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" path="/var/lib/kubelet/pods/ac669eb9-9135-44df-858a-7fd3c7238e28/volumes" Nov 03 22:36:03.138578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.138564 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" path="/var/lib/kubelet/pods/d8c87f27-faa4-4ce3-8c24-939548b06801/volumes" Nov 03 22:36:03.139016 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.138997 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debadcac-80e7-4843-a84a-dc60f364566c" path="/var/lib/kubelet/pods/debadcac-80e7-4843-a84a-dc60f364566c/volumes" Nov 03 22:36:03.139719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.139704 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" path="/var/lib/kubelet/pods/e55d3001-b0d1-41a6-a376-8f057710801b/volumes" Nov 03 22:36:03.201702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.201667 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t"] Nov 03 22:36:03.206302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.206268 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-6b25t"] Nov 03 22:36:03.833687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:03.833657 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:04.088365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.088193 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:04.175510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.175469 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9"] Nov 03 22:36:04.176407 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176376 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" Nov 03 22:36:04.176499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176410 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" Nov 03 22:36:04.176499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176431 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerName="istio-proxy" Nov 03 22:36:04.176499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176439 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerName="istio-proxy" Nov 03 22:36:04.176499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176472 4298 cpu_manager.go:414] "containerMap: removing container" podUID="8582c1e0-3305-4331-9a89-6aa46a71ae1b" containerName="revoke-keys" Nov 03 22:36:04.176499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176482 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8582c1e0-3305-4331-9a89-6aa46a71ae1b" containerName="revoke-keys" Nov 03 22:36:04.176499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176496 4298 cpu_manager.go:414] "containerMap: removing container" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176504 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176524 4298 cpu_manager.go:414] "containerMap: removing container" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176531 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176550 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176556 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176564 4298 cpu_manager.go:414] "containerMap: removing container" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176570 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176582 4298 cpu_manager.go:414] "containerMap: removing container" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176589 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176604 4298 cpu_manager.go:414] "containerMap: removing container" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176610 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176783 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="914760e1-f094-48ab-848f-e0e168fb265e" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176808 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="debadcac-80e7-4843-a84a-dc60f364566c" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176817 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55d3001-b0d1-41a6-a376-8f057710801b" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176830 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176839 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="992ba108-76e1-45ee-b87b-7fe5dcd799b5" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176853 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c87f27-faa4-4ce3-8c24-939548b06801" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176867 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac669eb9-9135-44df-858a-7fd3c7238e28" containerName="istio-proxy" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176880 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="8582c1e0-3305-4331-9a89-6aa46a71ae1b" containerName="revoke-keys" Nov 03 22:36:04.176929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.176890 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2411c9-7385-4b74-b850-2006e89a8f54" containerName="istio-proxy" Nov 03 22:36:04.183457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.183435 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:36:04.235860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.235827 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f61dc0\"" Nov 03 22:36:04.236710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.236686 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9"] Nov 03 22:36:04.255889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.255850 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xd2\" (UniqueName: \"kubernetes.io/projected/2359d456-83bd-4d71-8b79-d13bde8ff0da-kube-api-access-v7xd2\") pod \"payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9\" (UID: \"2359d456-83bd-4d71-8b79-d13bde8ff0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:36:04.256069 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.255918 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2359d456-83bd-4d71-8b79-d13bde8ff0da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9\" (UID: \"2359d456-83bd-4d71-8b79-d13bde8ff0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:36:04.327850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.326527 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db"] Nov 03 22:36:04.335543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.335511 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:36:04.357384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.357354 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2359d456-83bd-4d71-8b79-d13bde8ff0da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9\" (UID: \"2359d456-83bd-4d71-8b79-d13bde8ff0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:36:04.357593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.357576 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xd2\" (UniqueName: \"kubernetes.io/projected/2359d456-83bd-4d71-8b79-d13bde8ff0da-kube-api-access-v7xd2\") pod \"payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9\" (UID: \"2359d456-83bd-4d71-8b79-d13bde8ff0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:36:04.358267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.358244 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2359d456-83bd-4d71-8b79-d13bde8ff0da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9\" (UID: \"2359d456-83bd-4d71-8b79-d13bde8ff0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:36:04.372913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.372883 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db"] Nov 03 22:36:04.399558 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.397799 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f61dc0-dockercfg-8p74n\"" Nov 03 22:36:04.418674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.416185 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xd2\" (UniqueName: \"kubernetes.io/projected/2359d456-83bd-4d71-8b79-d13bde8ff0da-kube-api-access-v7xd2\") pod \"payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9\" (UID: \"2359d456-83bd-4d71-8b79-d13bde8ff0da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:36:04.460816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.460776 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db\" (UID: \"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:36:04.462956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.462923 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2mq\" (UniqueName: \"kubernetes.io/projected/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-kube-api-access-wv2mq\") pod \"payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db\" (UID: \"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:36:04.499208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.499177 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:36:04.566413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.565138 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db\" (UID: \"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:36:04.566413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.565391 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2mq\" (UniqueName: \"kubernetes.io/projected/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-kube-api-access-wv2mq\") pod \"payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db\" (UID: \"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:36:04.567018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.566798 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db\" (UID: \"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:36:04.624614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.623496 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2mq\" (UniqueName: \"kubernetes.io/projected/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-kube-api-access-wv2mq\") pod \"payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db\" (UID: \"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:36:04.652695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.649919 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:04.654917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.654879 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:36:04.889311 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:36:04.889267 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2359d456_83bd_4d71_8b79_d13bde8ff0da.slice/crio-6f67af5122d17c54ecce7f8ea47aa519f237cdc4a6609b6b744ac6b04e62f150 WatchSource:0}: Error finding container 6f67af5122d17c54ecce7f8ea47aa519f237cdc4a6609b6b744ac6b04e62f150: Status 404 returned error can't find the container with id 6f67af5122d17c54ecce7f8ea47aa519f237cdc4a6609b6b744ac6b04e62f150 Nov 03 22:36:04.891485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:04.891460 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9"] Nov 03 22:36:05.065146 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:36:05.065110 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83dc4bb3_9757_4aee_9cbf_f6e2eb3d03de.slice/crio-097689c795f978a2d7b1a8e56620010f12ec465cc8620696399a2b0bc26764dd WatchSource:0}: Error finding container 097689c795f978a2d7b1a8e56620010f12ec465cc8620696399a2b0bc26764dd: Status 404 returned error can't find the container with id 097689c795f978a2d7b1a8e56620010f12ec465cc8620696399a2b0bc26764dd Nov 03 22:36:05.069813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:05.069786 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db"] Nov 03 22:36:05.138158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:05.138126 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a156fac-b1b4-4cf1-a546-658009fec833" path="/var/lib/kubelet/pods/9a156fac-b1b4-4cf1-a546-658009fec833/volumes" Nov 03 22:36:05.218570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:05.218541 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:05.845715 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:05.845676 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" event={"ID":"2359d456-83bd-4d71-8b79-d13bde8ff0da","Type":"ContainerStarted","Data":"d7931fe1cb02e99e1521b45d77ae091ab7dad3d4e639e406aa760d962c1d386a"} Nov 03 22:36:05.845908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:05.845732 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" event={"ID":"2359d456-83bd-4d71-8b79-d13bde8ff0da","Type":"ContainerStarted","Data":"6f67af5122d17c54ecce7f8ea47aa519f237cdc4a6609b6b744ac6b04e62f150"} Nov 03 22:36:05.845908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:05.845890 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:36:05.847472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:05.847444 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" event={"ID":"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de","Type":"ContainerStarted","Data":"78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2"} Nov 03 22:36:05.847608 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:05.847492 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" event={"ID":"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de","Type":"ContainerStarted","Data":"097689c795f978a2d7b1a8e56620010f12ec465cc8620696399a2b0bc26764dd"} Nov 03 22:36:05.847608 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:05.847518 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:36:05.965005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:05.964958 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" podStartSLOduration=1.96494489 podStartE2EDuration="1.96494489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:05.964794741 +0000 UTC m=+1722.945528393" watchObservedRunningTime="2026-11-03 22:36:05.96494489 +0000 UTC m=+1722.945678555" Nov 03 22:36:06.083173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:06.083117 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" podStartSLOduration=2.083103818 podStartE2EDuration="2.083103818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:06.078047376 +0000 UTC m=+1723.058781028" watchObservedRunningTime="2026-11-03 22:36:06.083103818 +0000 UTC m=+1723.063837471" Nov 03 22:36:09.906566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:09.906511 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:12.809011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:12.808982 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:14.552227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:14.552187 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:16.861914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:16.861869 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:36:16.862412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:16.861965 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:36:17.101956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.101920 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48"] Nov 03 22:36:17.106348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.106316 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:17.158966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.158936 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:36:17.159165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.159098 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:36:17.161114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.161083 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48"] Nov 03 22:36:17.161831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.161801 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee9c090c-7tv48\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:17.162287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.162252 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqjlt\" (UniqueName: \"kubernetes.io/projected/ed014b7f-5354-4201-b2b5-df5bd0e81c05-kube-api-access-lqjlt\") pod \"maas-api-revoke-sub-ee9c090c-7tv48\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:17.162400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.162322 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee9c090c-7tv48\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:17.263670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.263609 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee9c090c-7tv48\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:17.263897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.263878 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqjlt\" (UniqueName: \"kubernetes.io/projected/ed014b7f-5354-4201-b2b5-df5bd0e81c05-kube-api-access-lqjlt\") pod \"maas-api-revoke-sub-ee9c090c-7tv48\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:17.263971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.263923 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee9c090c-7tv48\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:17.264399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.264366 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee9c090c-7tv48\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:17.266303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.266280 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee9c090c-7tv48\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:17.320797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.320767 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqjlt\" (UniqueName: \"kubernetes.io/projected/ed014b7f-5354-4201-b2b5-df5bd0e81c05-kube-api-access-lqjlt\") pod \"maas-api-revoke-sub-ee9c090c-7tv48\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:17.426806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.426714 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:17.565739 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:17.565699 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:36:17.799637 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:36:17.799599 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded014b7f_5354_4201_b2b5_df5bd0e81c05.slice/crio-af6f7cce46fed938ee1fbc1280410d4df7d70cc078cc844c948a6fbb408585b3 WatchSource:0}: Error finding container af6f7cce46fed938ee1fbc1280410d4df7d70cc078cc844c948a6fbb408585b3: Status 404 returned error can't find the container with id af6f7cce46fed938ee1fbc1280410d4df7d70cc078cc844c948a6fbb408585b3 Nov 03 22:36:17.804043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.804012 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48"] Nov 03 22:36:17.953168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:17.953131 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" event={"ID":"ed014b7f-5354-4201-b2b5-df5bd0e81c05","Type":"ContainerStarted","Data":"af6f7cce46fed938ee1fbc1280410d4df7d70cc078cc844c948a6fbb408585b3"} Nov 03 22:36:18.961232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:18.961188 4298 generic.go:356] "Generic (PLEG): container finished" podID="ed014b7f-5354-4201-b2b5-df5bd0e81c05" containerID="445ca4a47b562127ccf59fc2352d42e8f77203e9efc5b28129080acd56346acf" exitCode=0 Nov 03 22:36:18.961725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:18.961274 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" event={"ID":"ed014b7f-5354-4201-b2b5-df5bd0e81c05","Type":"ContainerDied","Data":"445ca4a47b562127ccf59fc2352d42e8f77203e9efc5b28129080acd56346acf"} Nov 03 22:36:19.077989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:19.077938 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:20.129280 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.129255 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:20.205876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.205843 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed014b7f-5354-4201-b2b5-df5bd0e81c05-kube-api-access-lqjlt\" (UniqueName: \"kubernetes.io/projected/ed014b7f-5354-4201-b2b5-df5bd0e81c05-kube-api-access-lqjlt\") pod \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " Nov 03 22:36:20.206082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.205921 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-xks-ca\") pod \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " Nov 03 22:36:20.206082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.205996 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-openshift-ca\") pod \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\" (UID: \"ed014b7f-5354-4201-b2b5-df5bd0e81c05\") " Nov 03 22:36:20.206353 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.206327 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-openshift-ca" pod "ed014b7f-5354-4201-b2b5-df5bd0e81c05" (UID: "ed014b7f-5354-4201-b2b5-df5bd0e81c05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:20.206543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.206523 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:20.207975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.207954 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-xks-ca" pod "ed014b7f-5354-4201-b2b5-df5bd0e81c05" (UID: "ed014b7f-5354-4201-b2b5-df5bd0e81c05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:36:20.208118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.208098 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed014b7f-5354-4201-b2b5-df5bd0e81c05-kube-api-access-lqjlt" pod "ed014b7f-5354-4201-b2b5-df5bd0e81c05" (UID: "ed014b7f-5354-4201-b2b5-df5bd0e81c05"). InnerVolumeSpecName "kube-api-access-lqjlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:20.308525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.308436 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqjlt\" (UniqueName: \"kubernetes.io/projected/ed014b7f-5354-4201-b2b5-df5bd0e81c05-kube-api-access-lqjlt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:20.308525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.308469 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed014b7f-5354-4201-b2b5-df5bd0e81c05-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:20.975863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.975833 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" Nov 03 22:36:20.976057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.975873 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48" event={"ID":"ed014b7f-5354-4201-b2b5-df5bd0e81c05","Type":"ContainerDied","Data":"af6f7cce46fed938ee1fbc1280410d4df7d70cc078cc844c948a6fbb408585b3"} Nov 03 22:36:20.976057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:20.975928 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af6f7cce46fed938ee1fbc1280410d4df7d70cc078cc844c948a6fbb408585b3" Nov 03 22:36:23.514697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:23.514668 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:24.760938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:24.760901 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924"] Nov 03 22:36:24.761403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:24.761136 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" containerName="istio-proxy" containerID="cri-o://cad9cfe1c5a0da8bbea8d0a4c4521af0313527b387c2f34941702c5bec5e4ec9" gracePeriod=30 Nov 03 22:36:24.811543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:24.811512 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7"] Nov 03 22:36:24.811773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:24.811735 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" containerID="cri-o://63a92d0dde8f2308ffba3fab046a0ef8af5d35a981109f97b1943149a56ece83" gracePeriod=30 Nov 03 22:36:25.112478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:25.112390 4298 scope.go:122] "RemoveContainer" containerID="9cd7f1f6548eca7b6d97f6d48eafc0c31aafc04a9115e4c728a515295ab01641" Nov 03 22:36:25.122962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:25.122936 4298 scope.go:122] "RemoveContainer" containerID="d8b50a32e19de10461cff6be75e522e24d9a77301e499b086f90ee69cd335215" Nov 03 22:36:25.140113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:25.140089 4298 scope.go:122] "RemoveContainer" containerID="ba388dae3773e1bf2c328df1fc48b8734d0d88fa583d9f6232207595526e3564" Nov 03 22:36:25.149548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:25.149526 4298 scope.go:122] "RemoveContainer" containerID="9122ae334a2886aebc35c6954b87a7de3f190d0be5347a25dc0c2e76a9116469" Nov 03 22:36:25.158588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:25.158570 4298 scope.go:122] "RemoveContainer" containerID="0a144e5f2a6d56f481e2f6a520ea9d2ff6853479b97980fe54b0d1f7e96ff182" Nov 03 22:36:25.167298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:25.167280 4298 scope.go:122] "RemoveContainer" containerID="168d7c448cf65b3d9066915189c4ffbbb801b2bbd1d37bfce9fd0be06403900c" Nov 03 22:36:25.176251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:25.176232 4298 scope.go:122] "RemoveContainer" containerID="bccf2eb16385e75d427f433f2f566e2814056e7886bea0e4a228f9265e68a105" Nov 03 22:36:25.594521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:25.594483 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld"] Nov 03 22:36:25.594811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:25.594699 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" containerID="cri-o://d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e" gracePeriod=30 Nov 03 22:36:27.818199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:27.818161 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:29.107769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:29.107735 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:30.053750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.053714 4298 generic.go:356] "Generic (PLEG): container finished" podID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerID="63a92d0dde8f2308ffba3fab046a0ef8af5d35a981109f97b1943149a56ece83" exitCode=0 Nov 03 22:36:30.053934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.053895 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" event={"ID":"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7","Type":"ContainerDied","Data":"63a92d0dde8f2308ffba3fab046a0ef8af5d35a981109f97b1943149a56ece83"} Nov 03 22:36:30.055978 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.055945 4298 generic.go:356] "Generic (PLEG): container finished" podID="ea941933-edc9-4938-b3e6-42caa269c62b" containerID="cad9cfe1c5a0da8bbea8d0a4c4521af0313527b387c2f34941702c5bec5e4ec9" exitCode=0 Nov 03 22:36:30.056104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.056024 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" event={"ID":"ea941933-edc9-4938-b3e6-42caa269c62b","Type":"ContainerDied","Data":"cad9cfe1c5a0da8bbea8d0a4c4521af0313527b387c2f34941702c5bec5e4ec9"} Nov 03 22:36:30.096128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.096102 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:36:30.128909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.128874 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:36:30.247577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.247537 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-certs\") pod \"ea941933-edc9-4938-b3e6-42caa269c62b\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " Nov 03 22:36:30.247817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.247623 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-istio-token\" (UniqueName: \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-istio-token\") pod \"ea941933-edc9-4938-b3e6-42caa269c62b\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " Nov 03 22:36:30.247817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.247678 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-podinfo\") pod \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " Nov 03 22:36:30.247817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.247707 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-socket\") pod \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " Nov 03 22:36:30.247817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.247743 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istiod-ca-cert\") pod \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " Nov 03 22:36:30.247817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.247804 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-credential-socket\") pod \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " Nov 03 22:36:30.248173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.247911 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-certs" pod "ea941933-edc9-4938-b3e6-42caa269c62b" (UID: "ea941933-edc9-4938-b3e6-42caa269c62b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:30.248173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.247949 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-kube-api-access-6rtz2\" (UniqueName: \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-kube-api-access-6rtz2\") pod \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " Nov 03 22:36:30.248173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.247970 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea941933-edc9-4938-b3e6-42caa269c62b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea941933-edc9-4938-b3e6-42caa269c62b-istio-podinfo\") pod \"ea941933-edc9-4938-b3e6-42caa269c62b\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " Nov 03 22:36:30.248173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248037 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-socket\") pod \"ea941933-edc9-4938-b3e6-42caa269c62b\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " Nov 03 22:36:30.248173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248065 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-data\") pod \"ea941933-edc9-4938-b3e6-42caa269c62b\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " Nov 03 22:36:30.248173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248101 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-socket" pod "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" (UID: "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:30.248173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248139 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-credential-socket\") pod \"ea941933-edc9-4938-b3e6-42caa269c62b\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " Nov 03 22:36:30.248173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248160 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-credential-socket" pod "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" (UID: "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:30.248173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248168 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-envoy\") pod \"ea941933-edc9-4938-b3e6-42caa269c62b\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " Nov 03 22:36:30.248687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248207 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-certs\") pod \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " Nov 03 22:36:30.248687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248273 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-token\" (UniqueName: \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-token\") pod \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " Nov 03 22:36:30.248687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248313 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-kube-api-access-pjcnd\" (UniqueName: \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-kube-api-access-pjcnd\") pod \"ea941933-edc9-4938-b3e6-42caa269c62b\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " Nov 03 22:36:30.248687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248344 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-socket" pod "ea941933-edc9-4938-b3e6-42caa269c62b" (UID: "ea941933-edc9-4938-b3e6-42caa269c62b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:30.248687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248368 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-data" pod "ea941933-edc9-4938-b3e6-42caa269c62b" (UID: "ea941933-edc9-4938-b3e6-42caa269c62b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:30.248687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248388 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-envoy\") pod \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " Nov 03 22:36:30.248687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248438 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-data\") pod \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\" (UID: \"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7\") " Nov 03 22:36:30.248687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248487 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea941933-edc9-4938-b3e6-42caa269c62b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea941933-edc9-4938-b3e6-42caa269c62b-istiod-ca-cert\") pod \"ea941933-edc9-4938-b3e6-42caa269c62b\" (UID: \"ea941933-edc9-4938-b3e6-42caa269c62b\") " Nov 03 22:36:30.248687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248524 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istiod-ca-cert" pod "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" (UID: "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:30.249201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.248811 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-certs" pod "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" (UID: "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:30.249371 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.249349 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.249532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.249515 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.249655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.249628 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.249769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.249754 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.249867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.249855 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.249960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.249948 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.250090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.250076 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.250713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.249542 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-data" pod "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" (UID: "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:30.250841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.250032 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-credential-socket" pod "ea941933-edc9-4938-b3e6-42caa269c62b" (UID: "ea941933-edc9-4938-b3e6-42caa269c62b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:30.250841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.250231 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea941933-edc9-4938-b3e6-42caa269c62b-istiod-ca-cert" pod "ea941933-edc9-4938-b3e6-42caa269c62b" (UID: "ea941933-edc9-4938-b3e6-42caa269c62b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:30.250841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.250665 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-istio-token" pod "ea941933-edc9-4938-b3e6-42caa269c62b" (UID: "ea941933-edc9-4938-b3e6-42caa269c62b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:30.250841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.250818 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-podinfo" pod "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" (UID: "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:30.251068 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.250924 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-envoy" pod "ea941933-edc9-4938-b3e6-42caa269c62b" (UID: "ea941933-edc9-4938-b3e6-42caa269c62b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:30.251188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.251165 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-token" pod "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" (UID: "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:30.251248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.251201 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-kube-api-access-6rtz2" pod "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" (UID: "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7"). InnerVolumeSpecName "kube-api-access-6rtz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:30.251721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.251688 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea941933-edc9-4938-b3e6-42caa269c62b-istio-podinfo" pod "ea941933-edc9-4938-b3e6-42caa269c62b" (UID: "ea941933-edc9-4938-b3e6-42caa269c62b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:30.251827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.251808 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-kube-api-access-pjcnd" pod "ea941933-edc9-4938-b3e6-42caa269c62b" (UID: "ea941933-edc9-4938-b3e6-42caa269c62b"). InnerVolumeSpecName "kube-api-access-pjcnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:30.252440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.252421 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-envoy" pod "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" (UID: "fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:30.351699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.351595 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rtz2\" (UniqueName: \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-kube-api-access-6rtz2\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.351699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.351627 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea941933-edc9-4938-b3e6-42caa269c62b-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.351699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.351637 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.351699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.351668 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea941933-edc9-4938-b3e6-42caa269c62b-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.351699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.351679 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.351699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.351687 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjcnd\" (UniqueName: \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-kube-api-access-pjcnd\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.351699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.351695 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.351699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.351704 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.352084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.351713 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea941933-edc9-4938-b3e6-42caa269c62b-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.352084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.351721 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea941933-edc9-4938-b3e6-42caa269c62b-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.352084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.351732 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:30.369003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.368969 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n"] Nov 03 22:36:30.369534 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.369522 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ed014b7f-5354-4201-b2b5-df5bd0e81c05" containerName="revoke-keys" Nov 03 22:36:30.369583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.369537 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed014b7f-5354-4201-b2b5-df5bd0e81c05" containerName="revoke-keys" Nov 03 22:36:30.369583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.369557 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" containerName="istio-proxy" Nov 03 22:36:30.369583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.369562 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" containerName="istio-proxy" Nov 03 22:36:30.369583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.369583 4298 cpu_manager.go:414] "containerMap: removing container" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" Nov 03 22:36:30.369737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.369588 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" Nov 03 22:36:30.369737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.369710 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" containerName="istio-proxy" Nov 03 22:36:30.369737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.369724 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed014b7f-5354-4201-b2b5-df5bd0e81c05" containerName="revoke-keys" Nov 03 22:36:30.369737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.369734 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" containerName="istio-proxy" Nov 03 22:36:30.374503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.374482 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.381789 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.381768 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr"] Nov 03 22:36:30.385747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.385729 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.435580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.435545 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n"] Nov 03 22:36:30.487887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.487849 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr"] Nov 03 22:36:30.554505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.554469 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.554702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.554518 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.554702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.554596 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.554702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.554688 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlsxs\" (UniqueName: \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-kube-api-access-rlsxs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.554832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.554741 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.554876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.554853 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqrv\" (UniqueName: \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-kube-api-access-fsqrv\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.554923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.554885 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.555016 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.554997 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.555084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.555031 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.555084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.555064 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.555183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.555092 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.555183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.555121 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.555183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.555162 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.555298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.555200 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.555337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.555295 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.555337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.555324 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.555409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.555347 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.555409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.555378 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d46a0c-7111-487a-862b-a89a8bf4ecce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.657139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657099 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlsxs\" (UniqueName: \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-kube-api-access-rlsxs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.657318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657179 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.657318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657265 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqrv\" (UniqueName: \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-kube-api-access-fsqrv\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.657318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657304 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.657490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657371 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.657490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657397 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.657595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657503 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.657595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657534 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.657595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657562 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.657595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657587 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.657851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657612 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.657851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657682 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.657851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657711 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.657851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657747 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.657851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657804 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.657851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657804 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d46a0c-7111-487a-862b-a89a8bf4ecce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.658188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657845 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.658188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657883 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.658188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657938 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.658188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.657958 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.658188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.658013 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.658188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.658047 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.658188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.658157 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.658561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.658343 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.658561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.658367 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.658561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.658424 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.658561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.658554 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d46a0c-7111-487a-862b-a89a8bf4ecce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.659079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.659038 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.660340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.660309 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.660340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.660321 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.660463 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.660357 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.660508 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.660463 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.714771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.714735 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.714928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.714797 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlsxs\" (UniqueName: \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-kube-api-access-rlsxs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.715026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.715007 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-gkrnr\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:30.715085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.715013 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqrv\" (UniqueName: \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-kube-api-access-fsqrv\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6tj7n\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.875212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.875148 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:36:30.987980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.987949 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:30.998832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:30.998800 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:31.066585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.064257 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-certs\") pod \"19a3eb67-b858-4181-b161-8204932fe1a6\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " Nov 03 22:36:31.066585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.064304 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-istio-token\" (UniqueName: \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-istio-token\") pod \"19a3eb67-b858-4181-b161-8204932fe1a6\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " Nov 03 22:36:31.066585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.064357 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-socket\") pod \"19a3eb67-b858-4181-b161-8204932fe1a6\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " Nov 03 22:36:31.066585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.064469 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-kube-api-access-vvfln\" (UniqueName: \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-kube-api-access-vvfln\") pod \"19a3eb67-b858-4181-b161-8204932fe1a6\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " Nov 03 22:36:31.066585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.064530 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a3eb67-b858-4181-b161-8204932fe1a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a3eb67-b858-4181-b161-8204932fe1a6-istiod-ca-cert\") pod \"19a3eb67-b858-4181-b161-8204932fe1a6\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " Nov 03 22:36:31.066585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.064717 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-data\") pod \"19a3eb67-b858-4181-b161-8204932fe1a6\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " Nov 03 22:36:31.066585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.064765 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19a3eb67-b858-4181-b161-8204932fe1a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a3eb67-b858-4181-b161-8204932fe1a6-istio-podinfo\") pod \"19a3eb67-b858-4181-b161-8204932fe1a6\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " Nov 03 22:36:31.066585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.064830 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-envoy\") pod \"19a3eb67-b858-4181-b161-8204932fe1a6\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " Nov 03 22:36:31.066585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.064875 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-credential-socket\") pod \"19a3eb67-b858-4181-b161-8204932fe1a6\" (UID: \"19a3eb67-b858-4181-b161-8204932fe1a6\") " Nov 03 22:36:31.067685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.067299 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-certs" pod "19a3eb67-b858-4181-b161-8204932fe1a6" (UID: "19a3eb67-b858-4181-b161-8204932fe1a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:31.067685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.067474 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-socket" pod "19a3eb67-b858-4181-b161-8204932fe1a6" (UID: "19a3eb67-b858-4181-b161-8204932fe1a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:31.067887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.067864 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-data" pod "19a3eb67-b858-4181-b161-8204932fe1a6" (UID: "19a3eb67-b858-4181-b161-8204932fe1a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:31.068063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.068037 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a3eb67-b858-4181-b161-8204932fe1a6-istiod-ca-cert" pod "19a3eb67-b858-4181-b161-8204932fe1a6" (UID: "19a3eb67-b858-4181-b161-8204932fe1a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:31.068427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.068401 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-credential-socket" pod "19a3eb67-b858-4181-b161-8204932fe1a6" (UID: "19a3eb67-b858-4181-b161-8204932fe1a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:31.070833 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.070797 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19a3eb67-b858-4181-b161-8204932fe1a6-istio-podinfo" pod "19a3eb67-b858-4181-b161-8204932fe1a6" (UID: "19a3eb67-b858-4181-b161-8204932fe1a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:31.071067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.071042 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-kube-api-access-vvfln" pod "19a3eb67-b858-4181-b161-8204932fe1a6" (UID: "19a3eb67-b858-4181-b161-8204932fe1a6"). InnerVolumeSpecName "kube-api-access-vvfln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:31.082527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.081485 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" event={"ID":"ea941933-edc9-4938-b3e6-42caa269c62b","Type":"ContainerDied","Data":"de582f4f37a9ec778003e87a74369a002e8b8756a65df80e98a85a7b854dd6e4"} Nov 03 22:36:31.082527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.081563 4298 scope.go:122] "RemoveContainer" containerID="cad9cfe1c5a0da8bbea8d0a4c4521af0313527b387c2f34941702c5bec5e4ec9" Nov 03 22:36:31.082527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.081750 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924" Nov 03 22:36:31.087224 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.085402 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" event={"ID":"fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7","Type":"ContainerDied","Data":"6c86d49bad91bb73df77b506505423dceacc885abddecca41264ca2f77c8acc8"} Nov 03 22:36:31.087224 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.085523 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7" Nov 03 22:36:31.087224 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.086106 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-envoy" pod "19a3eb67-b858-4181-b161-8204932fe1a6" (UID: "19a3eb67-b858-4181-b161-8204932fe1a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:31.087224 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.086283 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-istio-token" pod "19a3eb67-b858-4181-b161-8204932fe1a6" (UID: "19a3eb67-b858-4181-b161-8204932fe1a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:31.088394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.087870 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" Nov 03 22:36:31.088394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.087629 4298 generic.go:356] "Generic (PLEG): container finished" podID="19a3eb67-b858-4181-b161-8204932fe1a6" containerID="d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e" exitCode=0 Nov 03 22:36:31.088394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.088323 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" event={"ID":"19a3eb67-b858-4181-b161-8204932fe1a6","Type":"ContainerDied","Data":"d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e"} Nov 03 22:36:31.088394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.088355 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld" event={"ID":"19a3eb67-b858-4181-b161-8204932fe1a6","Type":"ContainerDied","Data":"b6520d6296e1c9cb7a4a59adebc063b7b1efdea5d769abf32c35b40403e4c2bd"} Nov 03 22:36:31.120129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.105888 4298 scope.go:122] "RemoveContainer" containerID="63a92d0dde8f2308ffba3fab046a0ef8af5d35a981109f97b1943149a56ece83" Nov 03 22:36:31.133819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.133795 4298 scope.go:122] "RemoveContainer" containerID="d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e" Nov 03 22:36:31.152125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.152089 4298 scope.go:122] "RemoveContainer" containerID="d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e" Nov 03 22:36:31.152601 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:31.152569 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e\": container with ID starting with d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e not found: ID does not exist" containerID="d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e" Nov 03 22:36:31.152808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.152605 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e"} err="failed to get container status \"d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e\": rpc error: code = NotFound desc = could not find container \"d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e\": container with ID starting with d4e338b6534fcf6f25963e302501237fdff0f6c404c138c6e02721f86be3aa2e not found: ID does not exist" Nov 03 22:36:31.171341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.171311 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:31.171599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.171579 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a3eb67-b858-4181-b161-8204932fe1a6-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:31.171733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.171719 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:31.171891 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.171875 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:31.171997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.171984 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:31.172089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.172078 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:31.172201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.172189 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a3eb67-b858-4181-b161-8204932fe1a6-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:31.172297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.172283 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvfln\" (UniqueName: \"kubernetes.io/projected/19a3eb67-b858-4181-b161-8204932fe1a6-kube-api-access-vvfln\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:31.172393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.172369 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a3eb67-b858-4181-b161-8204932fe1a6-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:31.252929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.252897 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924"] Nov 03 22:36:31.257505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.257480 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7g924"] Nov 03 22:36:31.365770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.365738 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7"] Nov 03 22:36:31.370153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.370131 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-dfhp7"] Nov 03 22:36:31.406881 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:36:31.406849 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d46a0c_7111_487a_862b_a89a8bf4ecce.slice/crio-fb4f9eff3fed00f5af7fa6168d84ae93f4e64d280730fec3a6ef392fb0f30236 WatchSource:0}: Error finding container fb4f9eff3fed00f5af7fa6168d84ae93f4e64d280730fec3a6ef392fb0f30236: Status 404 returned error can't find the container with id fb4f9eff3fed00f5af7fa6168d84ae93f4e64d280730fec3a6ef392fb0f30236 Nov 03 22:36:31.410060 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.410021 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:36:31.410170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.410102 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:36:31.410170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.410142 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:36:31.412082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.412063 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n"] Nov 03 22:36:31.428123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.428095 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr"] Nov 03 22:36:31.428932 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:36:31.428908 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0aca7b_d296_4cde_9146_d4a0d30c7c40.slice/crio-6dfb52409d9448e2c23a01052816d76a43f63d18d9a5efbdc4c33f1cb2251c37 WatchSource:0}: Error finding container 6dfb52409d9448e2c23a01052816d76a43f63d18d9a5efbdc4c33f1cb2251c37: Status 404 returned error can't find the container with id 6dfb52409d9448e2c23a01052816d76a43f63d18d9a5efbdc4c33f1cb2251c37 Nov 03 22:36:31.431932 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.431901 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:36:31.432015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.431964 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:36:31.432015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.431995 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:36:31.478895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.478858 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld"] Nov 03 22:36:31.483982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:31.483958 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-5zgld"] Nov 03 22:36:32.096707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:32.096614 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" event={"ID":"2f0aca7b-d296-4cde-9146-d4a0d30c7c40","Type":"ContainerStarted","Data":"8b8846f20629bf353fc0e0215e624b2ba5daa00a0ce17a7ff6d1bc34bc997be1"} Nov 03 22:36:32.096949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:32.096930 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" event={"ID":"2f0aca7b-d296-4cde-9146-d4a0d30c7c40","Type":"ContainerStarted","Data":"6dfb52409d9448e2c23a01052816d76a43f63d18d9a5efbdc4c33f1cb2251c37"} Nov 03 22:36:32.099153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:32.099121 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" event={"ID":"00d46a0c-7111-487a-862b-a89a8bf4ecce","Type":"ContainerStarted","Data":"adaf6d6f50909ba3777020012f453d72a1d8347d4988e9cedcb0d5946e1e769c"} Nov 03 22:36:32.099272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:32.099170 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" event={"ID":"00d46a0c-7111-487a-862b-a89a8bf4ecce","Type":"ContainerStarted","Data":"fb4f9eff3fed00f5af7fa6168d84ae93f4e64d280730fec3a6ef392fb0f30236"} Nov 03 22:36:32.221114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:32.221058 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" podStartSLOduration=2.221043845 podStartE2EDuration="2.221043845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:32.21608742 +0000 UTC m=+1749.196821072" watchObservedRunningTime="2026-11-03 22:36:32.221043845 +0000 UTC m=+1749.201777498" Nov 03 22:36:32.409025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:32.408936 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:32.677718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:32.677333 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:32.988750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:32.988717 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:32.990117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:32.990086 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:36:32.990170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:32.990155 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 22:36:32.999295 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:32.999269 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:33.000436 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:33.000403 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:33.000512 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:33.000473 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 22:36:33.182524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:33.181145 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" path="/var/lib/kubelet/pods/19a3eb67-b858-4181-b161-8204932fe1a6/volumes" Nov 03 22:36:33.182524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:33.181663 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea941933-edc9-4938-b3e6-42caa269c62b" path="/var/lib/kubelet/pods/ea941933-edc9-4938-b3e6-42caa269c62b/volumes" Nov 03 22:36:33.182524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:33.182089 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7" path="/var/lib/kubelet/pods/fc4e9f4a-6f70-4f72-b2ee-efc803b54cc7/volumes" Nov 03 22:36:33.287864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:33.287751 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:33.988461 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:33.988424 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:36:33.988621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:33.988506 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 22:36:33.999976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:33.999950 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:34.000065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:34.000007 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 22:36:34.246721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:34.245661 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:34.988413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:34.988383 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:36:34.988884 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:34.988430 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 22:36:35.000215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:35.000188 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:35.000316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:35.000234 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 22:36:35.989287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:35.989250 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:36:35.989755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:35.989314 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 22:36:35.999918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:35.999889 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:36.000005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:35.999950 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 22:36:36.992114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:36.992078 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:36:36.992712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:36.992680 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 22:36:37.000261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:37.000232 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:37.000361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:37.000290 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 22:36:37.080067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:37.079184 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:37.270243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:37.270150 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:37.451140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:37.451102 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:37.988815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:37.988775 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:36:37.989014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:37.988829 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 22:36:38.000401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:38.000349 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:38.000912 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:38.000427 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 22:36:38.988913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:38.988861 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:36:38.989119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:38.988958 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 22:36:39.000518 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:39.000482 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:39.000947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:39.000549 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 22:36:40.270201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:40.270165 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:40.270658 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:40.270574 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:40.271905 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:40.271884 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:36:40.312190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:40.310405 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:40.312190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:40.312029 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:40.314007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:40.313961 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:36:40.439216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:40.439164 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" podStartSLOduration=10.439150669 podStartE2EDuration="10.439150669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:32.349035115 +0000 UTC m=+1749.329768772" watchObservedRunningTime="2026-11-03 22:36:40.439150669 +0000 UTC m=+1757.419884321" Nov 03 22:36:40.720155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:40.720118 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz"] Nov 03 22:36:40.720710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:40.720677 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" containerID="cri-o://b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918" gracePeriod=30 Nov 03 22:36:41.165376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:41.165298 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz"] Nov 03 22:36:41.165528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:41.165504 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" containerID="cri-o://2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9" gracePeriod=30 Nov 03 22:36:41.170478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:41.170451 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v"] Nov 03 22:36:41.170688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:41.170634 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" containerID="cri-o://062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575" gracePeriod=30 Nov 03 22:36:41.763635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:41.763600 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:44.541236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:44.541196 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:45.030976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.030940 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh"] Nov 03 22:36:45.031624 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.031606 4298 cpu_manager.go:414] "containerMap: removing container" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" Nov 03 22:36:45.031680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.031626 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" Nov 03 22:36:45.031754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.031743 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a3eb67-b858-4181-b161-8204932fe1a6" containerName="istio-proxy" Nov 03 22:36:45.035273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.035257 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.199693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.195994 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-fac3b6-openshift-default-dockercfg-gkggp\"" Nov 03 22:36:45.220902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.162633 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-envoy\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.221104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.220904 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-credential-socket\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.221104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.220993 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxwd\" (UniqueName: \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-kube-api-access-cqxwd\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.221104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.221065 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-data\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.221274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.221146 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-podinfo\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.221274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.221224 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istiod-ca-cert\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.221379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.221276 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-certs\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.221379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.221312 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-socket\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.223032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.221499 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-token\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.235615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.235585 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh"] Nov 03 22:36:45.236246 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.236204 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:45.322825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.322728 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-envoy\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.323132 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.323105 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-credential-socket\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.323261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.323188 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxwd\" (UniqueName: \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-kube-api-access-cqxwd\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.323326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.323259 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-data\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.323395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.323321 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-podinfo\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.323395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.323379 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istiod-ca-cert\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.323510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.323425 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-certs\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.323510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.323459 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-socket\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.323619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.323582 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-token\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.324405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.324373 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-credential-socket\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.325321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.324977 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-certs\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.325321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.325226 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-data\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.325321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.325262 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-socket\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.325552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.325449 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istiod-ca-cert\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.327882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.327848 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-envoy\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.328331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.328309 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-podinfo\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.381122 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.381085 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-token\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.381260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.381164 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxwd\" (UniqueName: \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-kube-api-access-cqxwd\") pod \"e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:45.663829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:45.663728 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:46.031355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.025304 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:36:46.033351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.033324 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:46.035562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.035532 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-data\") pod \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " Nov 03 22:36:46.035693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.035592 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-kube-api-access-dck7c\" (UniqueName: \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-kube-api-access-dck7c\") pod \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " Nov 03 22:36:46.035693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.035670 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-certs\") pod \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " Nov 03 22:36:46.035808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.035745 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-podinfo\") pod \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " Nov 03 22:36:46.035877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.035849 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-data" pod "43cfec8a-343e-42fc-bf6e-8dcc495c1220" (UID: "43cfec8a-343e-42fc-bf6e-8dcc495c1220"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.035950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.035875 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-credential-socket\") pod \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " Nov 03 22:36:46.035996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.035967 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-certs" pod "43cfec8a-343e-42fc-bf6e-8dcc495c1220" (UID: "43cfec8a-343e-42fc-bf6e-8dcc495c1220"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.036046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.036026 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istiod-ca-cert\") pod \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " Nov 03 22:36:46.036140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.036110 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-token\" (UniqueName: \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-token\") pod \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " Nov 03 22:36:46.036140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.036131 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-credential-socket" pod "43cfec8a-343e-42fc-bf6e-8dcc495c1220" (UID: "43cfec8a-343e-42fc-bf6e-8dcc495c1220"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.036252 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.036144 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-envoy\") pod \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " Nov 03 22:36:46.036252 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.036221 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-socket\") pod \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\" (UID: \"43cfec8a-343e-42fc-bf6e-8dcc495c1220\") " Nov 03 22:36:46.036453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.036418 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istiod-ca-cert" pod "43cfec8a-343e-42fc-bf6e-8dcc495c1220" (UID: "43cfec8a-343e-42fc-bf6e-8dcc495c1220"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:46.036747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.036723 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-socket" pod "43cfec8a-343e-42fc-bf6e-8dcc495c1220" (UID: "43cfec8a-343e-42fc-bf6e-8dcc495c1220"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.036850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.036823 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.036850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.036838 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.036850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.036848 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.037012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.036862 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.039105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.039080 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-kube-api-access-dck7c" pod "43cfec8a-343e-42fc-bf6e-8dcc495c1220" (UID: "43cfec8a-343e-42fc-bf6e-8dcc495c1220"). InnerVolumeSpecName "kube-api-access-dck7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:46.039816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.039785 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-podinfo" pod "43cfec8a-343e-42fc-bf6e-8dcc495c1220" (UID: "43cfec8a-343e-42fc-bf6e-8dcc495c1220"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:46.039816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.039799 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-envoy" pod "43cfec8a-343e-42fc-bf6e-8dcc495c1220" (UID: "43cfec8a-343e-42fc-bf6e-8dcc495c1220"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.039816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.039816 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-token" pod "43cfec8a-343e-42fc-bf6e-8dcc495c1220" (UID: "43cfec8a-343e-42fc-bf6e-8dcc495c1220"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:46.090857 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:36:46.090781 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef2fc6c_a501_4a62_adcf_879a017e7bd8.slice/crio-2955ec7f5cd3a6e062edf271a1ddc0dcd21053da60d425139071a4725c2ad4d5 WatchSource:0}: Error finding container 2955ec7f5cd3a6e062edf271a1ddc0dcd21053da60d425139071a4725c2ad4d5: Status 404 returned error can't find the container with id 2955ec7f5cd3a6e062edf271a1ddc0dcd21053da60d425139071a4725c2ad4d5 Nov 03 22:36:46.094023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.093983 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:36:46.094126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.094075 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:36:46.094126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.094118 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:36:46.095606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.095583 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh"] Nov 03 22:36:46.148235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.146718 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.148235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.146752 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dck7c\" (UniqueName: \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-kube-api-access-dck7c\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.148235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.146768 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.148235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.146784 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.148235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.146798 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfec8a-343e-42fc-bf6e-8dcc495c1220-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.284040 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.284005 4298 generic.go:356] "Generic (PLEG): container finished" podID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerID="b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918" exitCode=0 Nov 03 22:36:46.284282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.284264 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" event={"ID":"43cfec8a-343e-42fc-bf6e-8dcc495c1220","Type":"ContainerDied","Data":"b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918"} Nov 03 22:36:46.284372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.284315 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" event={"ID":"43cfec8a-343e-42fc-bf6e-8dcc495c1220","Type":"ContainerDied","Data":"6d3cd5198d4209bf721eeb0d935dc37e1e57d0088027d5b39c6925070b40cb17"} Nov 03 22:36:46.284372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.284345 4298 scope.go:122] "RemoveContainer" containerID="b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918" Nov 03 22:36:46.284517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.284504 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz" Nov 03 22:36:46.287060 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.287032 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" event={"ID":"1ef2fc6c-a501-4a62-adcf-879a017e7bd8","Type":"ContainerStarted","Data":"2955ec7f5cd3a6e062edf271a1ddc0dcd21053da60d425139071a4725c2ad4d5"} Nov 03 22:36:46.307184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.307042 4298 scope.go:122] "RemoveContainer" containerID="b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918" Nov 03 22:36:46.307732 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:46.307661 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918\": container with ID starting with b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918 not found: ID does not exist" containerID="b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918" Nov 03 22:36:46.307732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.307698 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918"} err="failed to get container status \"b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918\": rpc error: code = NotFound desc = could not find container \"b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918\": container with ID starting with b89c6afe6cac06a9c75c982be598cc63e33bf43343a16e66c93a57259c61d918 not found: ID does not exist" Nov 03 22:36:46.465021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.464976 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz"] Nov 03 22:36:46.477698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.474856 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-gxplz"] Nov 03 22:36:46.565581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.564214 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:46.576262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.575617 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:36:46.586732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.586254 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" podStartSLOduration=2.586236144 podStartE2EDuration="2.586236144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:46.583151853 +0000 UTC m=+1763.563885507" watchObservedRunningTime="2026-11-03 22:36:46.586236144 +0000 UTC m=+1763.566969798" Nov 03 22:36:46.639998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.639965 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:36:46.654851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.654817 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1728fe75-73a1-4f80-82d6-58b961ed5487-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1728fe75-73a1-4f80-82d6-58b961ed5487-istiod-ca-cert\") pod \"1728fe75-73a1-4f80-82d6-58b961ed5487\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " Nov 03 22:36:46.655029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.654860 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-credential-socket\") pod \"1728fe75-73a1-4f80-82d6-58b961ed5487\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " Nov 03 22:36:46.655029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.654905 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-podinfo\") pod \"1728fe75-73a1-4f80-82d6-58b961ed5487\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " Nov 03 22:36:46.655029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.654929 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-podinfo\") pod \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " Nov 03 22:36:46.655029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.654949 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-kube-api-access-g45bw\" (UniqueName: \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-kube-api-access-g45bw\") pod \"1728fe75-73a1-4f80-82d6-58b961ed5487\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " Nov 03 22:36:46.655029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.654994 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-envoy\") pod \"1728fe75-73a1-4f80-82d6-58b961ed5487\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " Nov 03 22:36:46.655029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655015 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-credential-socket\") pod \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " Nov 03 22:36:46.655434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655037 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-socket\") pod \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " Nov 03 22:36:46.655434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655060 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-data\") pod \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " Nov 03 22:36:46.655434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655105 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-kube-api-access-wrg78\" (UniqueName: \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-kube-api-access-wrg78\") pod \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " Nov 03 22:36:46.655434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655166 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-token\" (UniqueName: \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-token\") pod \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " Nov 03 22:36:46.655434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655209 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-socket\") pod \"1728fe75-73a1-4f80-82d6-58b961ed5487\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " Nov 03 22:36:46.655434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655237 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-envoy\") pod \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " Nov 03 22:36:46.655434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655270 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-certs\") pod \"1728fe75-73a1-4f80-82d6-58b961ed5487\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " Nov 03 22:36:46.655434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655269 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1728fe75-73a1-4f80-82d6-58b961ed5487-istiod-ca-cert" pod "1728fe75-73a1-4f80-82d6-58b961ed5487" (UID: "1728fe75-73a1-4f80-82d6-58b961ed5487"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:46.655434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655325 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-credential-socket" pod "4986afe5-cdd2-45d7-8012-bcbe881af8f3" (UID: "4986afe5-cdd2-45d7-8012-bcbe881af8f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.655434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655402 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-socket" pod "4986afe5-cdd2-45d7-8012-bcbe881af8f3" (UID: "4986afe5-cdd2-45d7-8012-bcbe881af8f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.655982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655292 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-certs\") pod \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " Nov 03 22:36:46.655982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655463 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-certs" pod "4986afe5-cdd2-45d7-8012-bcbe881af8f3" (UID: "4986afe5-cdd2-45d7-8012-bcbe881af8f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.655982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655496 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istiod-ca-cert\") pod \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\" (UID: \"4986afe5-cdd2-45d7-8012-bcbe881af8f3\") " Nov 03 22:36:46.655982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655543 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-token\" (UniqueName: \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-token\") pod \"1728fe75-73a1-4f80-82d6-58b961ed5487\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " Nov 03 22:36:46.655982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655584 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-data\") pod \"1728fe75-73a1-4f80-82d6-58b961ed5487\" (UID: \"1728fe75-73a1-4f80-82d6-58b961ed5487\") " Nov 03 22:36:46.655982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655602 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-data" pod "4986afe5-cdd2-45d7-8012-bcbe881af8f3" (UID: "4986afe5-cdd2-45d7-8012-bcbe881af8f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.655982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.655963 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-credential-socket" pod "1728fe75-73a1-4f80-82d6-58b961ed5487" (UID: "1728fe75-73a1-4f80-82d6-58b961ed5487"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.656320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.656181 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-data" pod "1728fe75-73a1-4f80-82d6-58b961ed5487" (UID: "1728fe75-73a1-4f80-82d6-58b961ed5487"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.656375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.656351 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.656375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.656371 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.656495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.656396 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1728fe75-73a1-4f80-82d6-58b961ed5487-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.656495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.656411 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.656495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.656425 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.656495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.656439 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.656495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.656452 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.656495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.656367 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-certs" pod "1728fe75-73a1-4f80-82d6-58b961ed5487" (UID: "1728fe75-73a1-4f80-82d6-58b961ed5487"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.656833 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.656542 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-socket" pod "1728fe75-73a1-4f80-82d6-58b961ed5487" (UID: "1728fe75-73a1-4f80-82d6-58b961ed5487"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.656833 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.656786 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istiod-ca-cert" pod "4986afe5-cdd2-45d7-8012-bcbe881af8f3" (UID: "4986afe5-cdd2-45d7-8012-bcbe881af8f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:46.658434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.658405 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-token" pod "4986afe5-cdd2-45d7-8012-bcbe881af8f3" (UID: "4986afe5-cdd2-45d7-8012-bcbe881af8f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:46.658589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.658513 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-kube-api-access-g45bw" pod "1728fe75-73a1-4f80-82d6-58b961ed5487" (UID: "1728fe75-73a1-4f80-82d6-58b961ed5487"). InnerVolumeSpecName "kube-api-access-g45bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:46.658589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.658532 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-podinfo" pod "4986afe5-cdd2-45d7-8012-bcbe881af8f3" (UID: "4986afe5-cdd2-45d7-8012-bcbe881af8f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:46.658923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.658902 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-envoy" pod "4986afe5-cdd2-45d7-8012-bcbe881af8f3" (UID: "4986afe5-cdd2-45d7-8012-bcbe881af8f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.659078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.659032 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-podinfo" pod "1728fe75-73a1-4f80-82d6-58b961ed5487" (UID: "1728fe75-73a1-4f80-82d6-58b961ed5487"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:46.659163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.659085 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-token" pod "1728fe75-73a1-4f80-82d6-58b961ed5487" (UID: "1728fe75-73a1-4f80-82d6-58b961ed5487"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:46.659345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.659320 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-kube-api-access-wrg78" pod "4986afe5-cdd2-45d7-8012-bcbe881af8f3" (UID: "4986afe5-cdd2-45d7-8012-bcbe881af8f3"). InnerVolumeSpecName "kube-api-access-wrg78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:46.660002 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.659982 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-envoy" pod "1728fe75-73a1-4f80-82d6-58b961ed5487" (UID: "1728fe75-73a1-4f80-82d6-58b961ed5487"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.665128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.665107 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:46.741519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.741461 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:46.761829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.761793 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.761829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.761828 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.762052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.761846 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.762052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.761860 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.762052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.761877 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.762052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.761911 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45bw\" (UniqueName: \"kubernetes.io/projected/1728fe75-73a1-4f80-82d6-58b961ed5487-kube-api-access-g45bw\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.762052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.761925 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.762052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.761939 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrg78\" (UniqueName: \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-kube-api-access-wrg78\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.762052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.761954 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.762052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.761968 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1728fe75-73a1-4f80-82d6-58b961ed5487-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.762052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.761983 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4986afe5-cdd2-45d7-8012-bcbe881af8f3-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.916933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:46.916829 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:47.138746 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.138713 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" path="/var/lib/kubelet/pods/43cfec8a-343e-42fc-bf6e-8dcc495c1220/volumes" Nov 03 22:36:47.295536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.295486 4298 generic.go:356] "Generic (PLEG): container finished" podID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerID="2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9" exitCode=0 Nov 03 22:36:47.295536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.295522 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" event={"ID":"1728fe75-73a1-4f80-82d6-58b961ed5487","Type":"ContainerDied","Data":"2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9"} Nov 03 22:36:47.295841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.295555 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" Nov 03 22:36:47.295841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.295585 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz" event={"ID":"1728fe75-73a1-4f80-82d6-58b961ed5487","Type":"ContainerDied","Data":"71765520efaf2ed45d04eca73de11b6914374e20c14d138f2f975642e33bd9c7"} Nov 03 22:36:47.295841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.295610 4298 scope.go:122] "RemoveContainer" containerID="2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9" Nov 03 22:36:47.298624 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.298602 4298 generic.go:356] "Generic (PLEG): container finished" podID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerID="062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575" exitCode=0 Nov 03 22:36:47.298858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.298840 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" event={"ID":"4986afe5-cdd2-45d7-8012-bcbe881af8f3","Type":"ContainerDied","Data":"062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575"} Nov 03 22:36:47.298946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.298880 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" event={"ID":"4986afe5-cdd2-45d7-8012-bcbe881af8f3","Type":"ContainerDied","Data":"cc41f9654cd05f7b5c487c90ce7e101bea382b97b3d53a68e704ea6f3c60f949"} Nov 03 22:36:47.299008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.298944 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v" Nov 03 22:36:47.301279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.301258 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" event={"ID":"1ef2fc6c-a501-4a62-adcf-879a017e7bd8","Type":"ContainerStarted","Data":"abe257aca0704ed70808099dd8dedbeb68c9e10d08f0f74fc55f4d744f03a57c"} Nov 03 22:36:47.307902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.307880 4298 scope.go:122] "RemoveContainer" containerID="2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9" Nov 03 22:36:47.308220 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:47.308202 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9\": container with ID starting with 2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9 not found: ID does not exist" containerID="2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9" Nov 03 22:36:47.308273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.308228 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9"} err="failed to get container status \"2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9\": rpc error: code = NotFound desc = could not find container \"2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9\": container with ID starting with 2941e081c428e0ac4ff2d580bdbbe673077d70c195f19b557e80f0d809b29bb9 not found: ID does not exist" Nov 03 22:36:47.308273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.308245 4298 scope.go:122] "RemoveContainer" containerID="062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575" Nov 03 22:36:47.317767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.317733 4298 scope.go:122] "RemoveContainer" containerID="062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575" Nov 03 22:36:47.318034 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:47.318016 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575\": container with ID starting with 062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575 not found: ID does not exist" containerID="062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575" Nov 03 22:36:47.318091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.318044 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575"} err="failed to get container status \"062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575\": rpc error: code = NotFound desc = could not find container \"062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575\": container with ID starting with 062e388cba7dcccd5d9d29f569f9db4caf10e99f07196409c9e5ae3df1b43575 not found: ID does not exist" Nov 03 22:36:47.470116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.470070 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz"] Nov 03 22:36:47.513964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.513924 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-glksz"] Nov 03 22:36:47.552942 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:36:47.552858 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:36:47.625120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.625077 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v"] Nov 03 22:36:47.629919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.629887 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-sdq9v"] Nov 03 22:36:47.671142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:47.671113 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:48.310587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:48.310558 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:48.311405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:48.311385 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:36:49.136592 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:49.136559 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" path="/var/lib/kubelet/pods/1728fe75-73a1-4f80-82d6-58b961ed5487/volumes" Nov 03 22:36:49.137077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:49.137063 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" path="/var/lib/kubelet/pods/4986afe5-cdd2-45d7-8012-bcbe881af8f3/volumes" Nov 03 22:36:49.557873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:49.557842 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:49.752724 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:49.752687 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:49.942630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:49.942546 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:51.531669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:51.531627 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:54.737926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:54.737894 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:55.258730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.258689 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj"] Nov 03 22:36:55.259293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.259265 4298 cpu_manager.go:414] "containerMap: removing container" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" Nov 03 22:36:55.259293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.259287 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" Nov 03 22:36:55.259462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.259329 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" Nov 03 22:36:55.259462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.259334 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" Nov 03 22:36:55.259462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.259354 4298 cpu_manager.go:414] "containerMap: removing container" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" Nov 03 22:36:55.259462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.259360 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" Nov 03 22:36:55.259577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.259518 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1728fe75-73a1-4f80-82d6-58b961ed5487" containerName="istio-proxy" Nov 03 22:36:55.259577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.259543 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="4986afe5-cdd2-45d7-8012-bcbe881af8f3" containerName="istio-proxy" Nov 03 22:36:55.259658 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.259627 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cfec8a-343e-42fc-bf6e-8dcc495c1220" containerName="istio-proxy" Nov 03 22:36:55.264605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.264586 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:36:55.296287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.296236 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfhtd\" (UniqueName: \"kubernetes.io/projected/82cb7348-5057-4da0-9c8d-fb00195e3ccc-kube-api-access-vfhtd\") pod \"payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj\" (UID: \"82cb7348-5057-4da0-9c8d-fb00195e3ccc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:36:55.296511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.296399 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cb7348-5057-4da0-9c8d-fb00195e3ccc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj\" (UID: \"82cb7348-5057-4da0-9c8d-fb00195e3ccc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:36:55.320778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.320743 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj"] Nov 03 22:36:55.365586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.365553 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-fac3b6-dockercfg-556m5\"" Nov 03 22:36:55.367322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.367298 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-fac3b6\"" Nov 03 22:36:55.397508 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.397474 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfhtd\" (UniqueName: \"kubernetes.io/projected/82cb7348-5057-4da0-9c8d-fb00195e3ccc-kube-api-access-vfhtd\") pod \"payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj\" (UID: \"82cb7348-5057-4da0-9c8d-fb00195e3ccc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:36:55.397685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.397558 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cb7348-5057-4da0-9c8d-fb00195e3ccc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj\" (UID: \"82cb7348-5057-4da0-9c8d-fb00195e3ccc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:36:55.398232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.398211 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cb7348-5057-4da0-9c8d-fb00195e3ccc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj\" (UID: \"82cb7348-5057-4da0-9c8d-fb00195e3ccc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:36:55.431362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.431308 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz"] Nov 03 22:36:55.437023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.436970 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:36:55.460710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.460675 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfhtd\" (UniqueName: \"kubernetes.io/projected/82cb7348-5057-4da0-9c8d-fb00195e3ccc-kube-api-access-vfhtd\") pod \"payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj\" (UID: \"82cb7348-5057-4da0-9c8d-fb00195e3ccc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:36:55.495250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.495212 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz"] Nov 03 22:36:55.499227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.499190 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjgm\" (UniqueName: \"kubernetes.io/projected/eea684a8-7762-45a7-8a60-a41d2e76cd4b-kube-api-access-mvjgm\") pod \"payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz\" (UID: \"eea684a8-7762-45a7-8a60-a41d2e76cd4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:36:55.499414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.499319 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea684a8-7762-45a7-8a60-a41d2e76cd4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz\" (UID: \"eea684a8-7762-45a7-8a60-a41d2e76cd4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:36:55.580026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.579931 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:36:55.600852 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.600819 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjgm\" (UniqueName: \"kubernetes.io/projected/eea684a8-7762-45a7-8a60-a41d2e76cd4b-kube-api-access-mvjgm\") pod \"payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz\" (UID: \"eea684a8-7762-45a7-8a60-a41d2e76cd4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:36:55.600986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.600959 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea684a8-7762-45a7-8a60-a41d2e76cd4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz\" (UID: \"eea684a8-7762-45a7-8a60-a41d2e76cd4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:36:55.601898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.601873 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea684a8-7762-45a7-8a60-a41d2e76cd4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz\" (UID: \"eea684a8-7762-45a7-8a60-a41d2e76cd4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:36:55.676344 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.676307 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjgm\" (UniqueName: \"kubernetes.io/projected/eea684a8-7762-45a7-8a60-a41d2e76cd4b-kube-api-access-mvjgm\") pod \"payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz\" (UID: \"eea684a8-7762-45a7-8a60-a41d2e76cd4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:36:55.759564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.754488 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:36:55.962797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:55.962773 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj"] Nov 03 22:36:56.160708 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:36:56.160586 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea684a8_7762_45a7_8a60_a41d2e76cd4b.slice/crio-d25ee62c4be26df7601a7ec815be3f88ccf7c517ae33e01e8df5f37424b062cc WatchSource:0}: Error finding container d25ee62c4be26df7601a7ec815be3f88ccf7c517ae33e01e8df5f37424b062cc: Status 404 returned error can't find the container with id d25ee62c4be26df7601a7ec815be3f88ccf7c517ae33e01e8df5f37424b062cc Nov 03 22:36:56.165333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:56.165297 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz"] Nov 03 22:36:56.369169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:56.369136 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" event={"ID":"eea684a8-7762-45a7-8a60-a41d2e76cd4b","Type":"ContainerStarted","Data":"36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89"} Nov 03 22:36:56.369377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:56.369196 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" event={"ID":"eea684a8-7762-45a7-8a60-a41d2e76cd4b","Type":"ContainerStarted","Data":"d25ee62c4be26df7601a7ec815be3f88ccf7c517ae33e01e8df5f37424b062cc"} Nov 03 22:36:56.369377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:56.369228 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:36:56.370782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:56.370752 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" event={"ID":"82cb7348-5057-4da0-9c8d-fb00195e3ccc","Type":"ContainerStarted","Data":"9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543"} Nov 03 22:36:56.370882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:56.370802 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" event={"ID":"82cb7348-5057-4da0-9c8d-fb00195e3ccc","Type":"ContainerStarted","Data":"723d3938ca92f367b5ffedecba5dd13050e12d0e3941d00f41d44ac5a0bda3ca"} Nov 03 22:36:56.370882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:56.370829 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:36:56.487925 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:56.487870 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" podStartSLOduration=1.4878555740000001 podStartE2EDuration="1.487855574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:56.481629719 +0000 UTC m=+1773.462363371" watchObservedRunningTime="2026-11-03 22:36:56.487855574 +0000 UTC m=+1773.468589252" Nov 03 22:36:56.603450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:56.599828 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" podStartSLOduration=1.599804073 podStartE2EDuration="1.599804073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:56.595140683 +0000 UTC m=+1773.575874338" watchObservedRunningTime="2026-11-03 22:36:56.599804073 +0000 UTC m=+1773.580537728" Nov 03 22:36:56.774779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:56.774606 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:58.201455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:58.201422 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:58.664967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:58.664872 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:59.155724 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.151591 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk"] Nov 03 22:36:59.160254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.160224 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.226670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.219874 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a92c9e-openshift-default-dockercfg-r4zqs\"" Nov 03 22:36:59.334728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.333074 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:59.339928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.337888 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-data\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.339928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.337931 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-certs\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.339928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.337951 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-credential-socket\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.339928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.338015 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-envoy\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.339928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.338035 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-podinfo\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.339928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.338056 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-token\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.339928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.338120 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istiod-ca-cert\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.339928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.338152 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-socket\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.339928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.338180 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdsb4\" (UniqueName: \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-kube-api-access-qdsb4\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.343801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.340738 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk"] Nov 03 22:36:59.440712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.439726 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-envoy\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.440712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.439775 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-podinfo\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.440712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.439810 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-token\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.441801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.441770 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istiod-ca-cert\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.441962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.441834 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-socket\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.441962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.441880 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdsb4\" (UniqueName: \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-kube-api-access-qdsb4\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.442084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.442073 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-data\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.442145 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.442113 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-certs\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.442204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.442144 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-credential-socket\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.442560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.442534 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-credential-socket\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.442839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.442737 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istiod-ca-cert\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.444728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.443289 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-certs\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.444728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.443521 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-data\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.444728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.443711 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-socket\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.458411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.447944 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-podinfo\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.474677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.473072 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-envoy\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.533156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.521681 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdsb4\" (UniqueName: \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-kube-api-access-qdsb4\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.533156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.522122 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-token\") pod \"e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:36:59.800807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:36:59.800772 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:37:00.057848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:00.057745 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:00.235801 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:37:00.235773 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c9d454_dbe9_4e6b_ab20_71b7388cedb0.slice/crio-a8e806485bf45dd4a9ebbcea06fab24b020bc0060700aba18ef67328c50ed598 WatchSource:0}: Error finding container a8e806485bf45dd4a9ebbcea06fab24b020bc0060700aba18ef67328c50ed598: Status 404 returned error can't find the container with id a8e806485bf45dd4a9ebbcea06fab24b020bc0060700aba18ef67328c50ed598 Nov 03 22:37:00.239493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:00.239448 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:37:00.239625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:00.239564 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:37:00.239625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:00.239612 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:37:00.241435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:00.241399 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk"] Nov 03 22:37:00.469667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:00.468634 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" event={"ID":"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0","Type":"ContainerStarted","Data":"f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee"} Nov 03 22:37:00.469667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:00.468732 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" event={"ID":"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0","Type":"ContainerStarted","Data":"a8e806485bf45dd4a9ebbcea06fab24b020bc0060700aba18ef67328c50ed598"} Nov 03 22:37:00.603726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:00.602584 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" podStartSLOduration=1.602565609 podStartE2EDuration="1.602565609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:00.596292083 +0000 UTC m=+1777.577025786" watchObservedRunningTime="2026-11-03 22:37:00.602565609 +0000 UTC m=+1777.583299263" Nov 03 22:37:00.801682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:00.801625 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:37:01.817668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:01.817248 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:37:02.537670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:02.535348 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:37:02.546681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:02.544958 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:37:02.829749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:02.826970 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:02.829749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:02.829464 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:02.975678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:02.956811 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:03.335021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:03.322462 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 22:37:03.413665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:03.411497 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 22:37:03.427669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:03.418573 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:03.921671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:03.913779 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:03.948670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:03.930838 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:04.013664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:04.013300 4298 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Nov 03 22:37:04.092916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:04.092879 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:07.177839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.177803 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:07.371059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.371023 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:07.381840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.381811 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:37:07.382026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.381886 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:37:07.686986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.686955 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n"] Nov 03 22:37:07.692591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.692570 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:37:07.756247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.756217 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-a92c9e\"" Nov 03 22:37:07.765720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.756630 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-a92c9e-dockercfg-qjl78\"" Nov 03 22:37:07.774802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.774589 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n"] Nov 03 22:37:07.816681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.781684 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n\" (UID: \"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:37:07.816990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.816753 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-682mg\" (UniqueName: \"kubernetes.io/projected/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-kube-api-access-682mg\") pod \"payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n\" (UID: \"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:37:07.873706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.873673 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx"] Nov 03 22:37:07.878193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.878171 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:37:07.918560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.918524 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n\" (UID: \"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:37:07.918748 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.918577 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-682mg\" (UniqueName: \"kubernetes.io/projected/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-kube-api-access-682mg\") pod \"payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n\" (UID: \"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:37:07.919269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.919248 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n\" (UID: \"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:37:07.937470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.937240 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx"] Nov 03 22:37:07.982459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:07.982428 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-682mg\" (UniqueName: \"kubernetes.io/projected/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-kube-api-access-682mg\") pod \"payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n\" (UID: \"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:37:08.007567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:08.007532 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:37:08.019992 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:08.019956 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx\" (UID: \"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:37:08.020261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:08.020242 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdh45\" (UniqueName: \"kubernetes.io/projected/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-kube-api-access-qdh45\") pod \"payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx\" (UID: \"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:37:08.121431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:08.121364 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx\" (UID: \"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:37:08.121737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:08.121716 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdh45\" (UniqueName: \"kubernetes.io/projected/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-kube-api-access-qdh45\") pod \"payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx\" (UID: \"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:37:08.122304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:08.122281 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx\" (UID: \"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:37:08.180685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:08.179223 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdh45\" (UniqueName: \"kubernetes.io/projected/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-kube-api-access-qdh45\") pod \"payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx\" (UID: \"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:37:08.192104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:08.192028 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:37:08.393436 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:37:08.393408 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178cb1ca_90dd_43b7_ae38_b2bbe401bd9c.slice/crio-d06e647858d9258197b4555def8557c6882486317805234fe930d249108985c8 WatchSource:0}: Error finding container d06e647858d9258197b4555def8557c6882486317805234fe930d249108985c8: Status 404 returned error can't find the container with id d06e647858d9258197b4555def8557c6882486317805234fe930d249108985c8 Nov 03 22:37:08.397227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:08.397141 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n"] Nov 03 22:37:08.570467 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:37:08.570432 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c647dcc_fd5b_42c5_a7a0_7eabd8b751c1.slice/crio-7e7b5b300a320ba92e0f3833d711989690b784a30d34b7f727fd78b6312285df WatchSource:0}: Error finding container 7e7b5b300a320ba92e0f3833d711989690b784a30d34b7f727fd78b6312285df: Status 404 returned error can't find the container with id 7e7b5b300a320ba92e0f3833d711989690b784a30d34b7f727fd78b6312285df Nov 03 22:37:08.574576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:08.574551 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx"] Nov 03 22:37:09.064866 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:09.064827 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" event={"ID":"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1","Type":"ContainerStarted","Data":"f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31"} Nov 03 22:37:09.065089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:09.064882 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" event={"ID":"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1","Type":"ContainerStarted","Data":"7e7b5b300a320ba92e0f3833d711989690b784a30d34b7f727fd78b6312285df"} Nov 03 22:37:09.065089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:09.064904 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:37:09.066589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:09.066563 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" event={"ID":"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c","Type":"ContainerStarted","Data":"c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc"} Nov 03 22:37:09.066746 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:09.066612 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" event={"ID":"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c","Type":"ContainerStarted","Data":"d06e647858d9258197b4555def8557c6882486317805234fe930d249108985c8"} Nov 03 22:37:09.066746 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:09.066697 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:37:09.185548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:09.185491 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" podStartSLOduration=2.185477444 podStartE2EDuration="2.185477444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:09.180311095 +0000 UTC m=+1786.161044748" watchObservedRunningTime="2026-11-03 22:37:09.185477444 +0000 UTC m=+1786.166211141" Nov 03 22:37:09.239109 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:09.239072 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:09.296164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:09.296097 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" podStartSLOduration=2.296078226 podStartE2EDuration="2.296078226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:09.294134022 +0000 UTC m=+1786.274867702" watchObservedRunningTime="2026-11-03 22:37:09.296078226 +0000 UTC m=+1786.276811880" Nov 03 22:37:13.167965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:13.167931 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:13.681105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:13.681072 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7094-071aa4"] Nov 03 22:37:13.685160 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:13.685141 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7094-071aa4" Nov 03 22:37:13.740897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:13.740864 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7094-071aa4"] Nov 03 22:37:13.786066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:13.786037 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jj8tk\"" Nov 03 22:37:13.801571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:13.801539 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbkf\" (UniqueName: \"kubernetes.io/projected/3ef69338-38cc-4030-933b-c9bb2e9048fc-kube-api-access-scbkf\") pod \"test-curl-7094-071aa4\" (UID: \"3ef69338-38cc-4030-933b-c9bb2e9048fc\") " pod="opendatahub/test-curl-7094-071aa4" Nov 03 22:37:13.902972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:13.902936 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbkf\" (UniqueName: \"kubernetes.io/projected/3ef69338-38cc-4030-933b-c9bb2e9048fc-kube-api-access-scbkf\") pod \"test-curl-7094-071aa4\" (UID: \"3ef69338-38cc-4030-933b-c9bb2e9048fc\") " pod="opendatahub/test-curl-7094-071aa4" Nov 03 22:37:13.959982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:13.959946 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbkf\" (UniqueName: \"kubernetes.io/projected/3ef69338-38cc-4030-933b-c9bb2e9048fc-kube-api-access-scbkf\") pod \"test-curl-7094-071aa4\" (UID: \"3ef69338-38cc-4030-933b-c9bb2e9048fc\") " pod="opendatahub/test-curl-7094-071aa4" Nov 03 22:37:13.998534 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:13.998497 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7094-071aa4" Nov 03 22:37:14.199123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:14.199091 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:14.364388 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:37:14.364350 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ef69338_38cc_4030_933b_c9bb2e9048fc.slice/crio-b268de2ba154f9818222e9d6f71e9e437d3ea10ae24efd918f5ed4a292f3cbbe WatchSource:0}: Error finding container b268de2ba154f9818222e9d6f71e9e437d3ea10ae24efd918f5ed4a292f3cbbe: Status 404 returned error can't find the container with id b268de2ba154f9818222e9d6f71e9e437d3ea10ae24efd918f5ed4a292f3cbbe Nov 03 22:37:14.368900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:14.368879 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7094-071aa4"] Nov 03 22:37:15.114895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:15.114857 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7094-071aa4" event={"ID":"3ef69338-38cc-4030-933b-c9bb2e9048fc","Type":"ContainerStarted","Data":"b268de2ba154f9818222e9d6f71e9e437d3ea10ae24efd918f5ed4a292f3cbbe"} Nov 03 22:37:16.126659 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:16.126614 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7094-071aa4" event={"ID":"3ef69338-38cc-4030-933b-c9bb2e9048fc","Type":"ContainerStarted","Data":"2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77"} Nov 03 22:37:16.241096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:16.241042 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7094-071aa4" podStartSLOduration=2.124256146 podStartE2EDuration="3.241028025s" totalImagesPullingTime="1.116771879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:16.238780851 +0000 UTC m=+1793.219514503" watchObservedRunningTime="2026-11-03 22:37:16.241028025 +0000 UTC m=+1793.221761678" Nov 03 22:37:17.553198 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:37:17.553158 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:37:17.745585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:17.745552 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7094-071aa4"] Nov 03 22:37:17.752634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:17.752602 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7094-071aa4"] Nov 03 22:37:18.141254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.141217 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7094-071aa4" podUID="3ef69338-38cc-4030-933b-c9bb2e9048fc" containerName="test-curl-7094-071aa4" containerID="cri-o://2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77" gracePeriod=2 Nov 03 22:37:18.193295 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:37:18.193259 4298 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7094-071aa4\" is forbidden: User \"system:node:ip-10-0-139-254.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-254.us-west-2.compute.internal' and this object" podUID="3ef69338-38cc-4030-933b-c9bb2e9048fc" pod="opendatahub/test-curl-7094-071aa4" Nov 03 22:37:18.653567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.653530 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz"] Nov 03 22:37:18.654237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.654220 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3ef69338-38cc-4030-933b-c9bb2e9048fc" containerName="test-curl-7094-071aa4" Nov 03 22:37:18.654277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.654238 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef69338-38cc-4030-933b-c9bb2e9048fc" containerName="test-curl-7094-071aa4" Nov 03 22:37:18.654362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.654351 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef69338-38cc-4030-933b-c9bb2e9048fc" containerName="test-curl-7094-071aa4" Nov 03 22:37:18.658018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.658001 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:18.709741 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:37:18.709688 4298 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7094-071aa4\" is forbidden: User \"system:node:ip-10-0-139-254.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-254.us-west-2.compute.internal' and this object" podUID="3ef69338-38cc-4030-933b-c9bb2e9048fc" pod="opendatahub/test-curl-7094-071aa4" Nov 03 22:37:18.714982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.714948 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz"] Nov 03 22:37:18.758902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.758863 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:18.759076 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.758956 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:37:18.761773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.761744 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:18.761915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.761893 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8qht\" (UniqueName: \"kubernetes.io/projected/ee489429-b3e4-4df7-ab1c-0ff2552cd802-kube-api-access-v8qht\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:18.762033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.762012 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:18.863723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.863679 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8qht\" (UniqueName: \"kubernetes.io/projected/ee489429-b3e4-4df7-ab1c-0ff2552cd802-kube-api-access-v8qht\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:18.863903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.863776 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:18.863903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.863883 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:18.864566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.864542 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:18.866200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.866180 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:18.920847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.920771 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8qht\" (UniqueName: \"kubernetes.io/projected/ee489429-b3e4-4df7-ab1c-0ff2552cd802-kube-api-access-v8qht\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:18.971264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:18.971225 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:19.354478 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:37:19.343600 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee489429_b3e4_4df7_ab1c_0ff2552cd802.slice/crio-52d826d46cb7e554881e301c4929b738a6097657027ddc65af3f50a86f19296a WatchSource:0}: Error finding container 52d826d46cb7e554881e301c4929b738a6097657027ddc65af3f50a86f19296a: Status 404 returned error can't find the container with id 52d826d46cb7e554881e301c4929b738a6097657027ddc65af3f50a86f19296a Nov 03 22:37:19.354478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:19.347446 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz"] Nov 03 22:37:19.660570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:19.660543 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:20.077535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:20.077509 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:37:20.077740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:20.077573 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:37:20.156345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:20.156312 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" event={"ID":"ee489429-b3e4-4df7-ab1c-0ff2552cd802","Type":"ContainerStarted","Data":"37f3f4eeb46ff68096d4edfe42134ade9b229a34e94d29b18170ffe9be9b578f"} Nov 03 22:37:20.156495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:20.156377 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" event={"ID":"ee489429-b3e4-4df7-ab1c-0ff2552cd802","Type":"ContainerStarted","Data":"52d826d46cb7e554881e301c4929b738a6097657027ddc65af3f50a86f19296a"} Nov 03 22:37:20.401352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:20.401322 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7094-071aa4" Nov 03 22:37:20.421399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:20.421339 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" podStartSLOduration=2.421325301 podStartE2EDuration="2.421325301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:20.417877296 +0000 UTC m=+1797.398610949" watchObservedRunningTime="2026-11-03 22:37:20.421325301 +0000 UTC m=+1797.402058953" Nov 03 22:37:20.487405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:20.487376 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef69338-38cc-4030-933b-c9bb2e9048fc-kube-api-access-scbkf\" (UniqueName: \"kubernetes.io/projected/3ef69338-38cc-4030-933b-c9bb2e9048fc-kube-api-access-scbkf\") pod \"3ef69338-38cc-4030-933b-c9bb2e9048fc\" (UID: \"3ef69338-38cc-4030-933b-c9bb2e9048fc\") " Nov 03 22:37:20.489472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:20.489444 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef69338-38cc-4030-933b-c9bb2e9048fc-kube-api-access-scbkf" pod "3ef69338-38cc-4030-933b-c9bb2e9048fc" (UID: "3ef69338-38cc-4030-933b-c9bb2e9048fc"). InnerVolumeSpecName "kube-api-access-scbkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:20.589051 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:20.589022 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scbkf\" (UniqueName: \"kubernetes.io/projected/3ef69338-38cc-4030-933b-c9bb2e9048fc-kube-api-access-scbkf\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:21.143955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:21.143877 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef69338-38cc-4030-933b-c9bb2e9048fc" path="/var/lib/kubelet/pods/3ef69338-38cc-4030-933b-c9bb2e9048fc/volumes" Nov 03 22:37:21.165453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:21.165417 4298 generic.go:356] "Generic (PLEG): container finished" podID="ee489429-b3e4-4df7-ab1c-0ff2552cd802" containerID="37f3f4eeb46ff68096d4edfe42134ade9b229a34e94d29b18170ffe9be9b578f" exitCode=0 Nov 03 22:37:21.165621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:21.165499 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" event={"ID":"ee489429-b3e4-4df7-ab1c-0ff2552cd802","Type":"ContainerDied","Data":"37f3f4eeb46ff68096d4edfe42134ade9b229a34e94d29b18170ffe9be9b578f"} Nov 03 22:37:21.166774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:21.166754 4298 generic.go:356] "Generic (PLEG): container finished" podID="3ef69338-38cc-4030-933b-c9bb2e9048fc" containerID="2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77" exitCode=137 Nov 03 22:37:21.166881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:21.166796 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7094-071aa4" Nov 03 22:37:21.166881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:21.166844 4298 scope.go:122] "RemoveContainer" containerID="2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77" Nov 03 22:37:21.176767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:21.176745 4298 scope.go:122] "RemoveContainer" containerID="2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77" Nov 03 22:37:21.177094 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:37:21.177074 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77\": container with ID starting with 2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77 not found: ID does not exist" containerID="2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77" Nov 03 22:37:21.177169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:21.177111 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77"} err="failed to get container status \"2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77\": rpc error: code = NotFound desc = could not find container \"2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77\": container with ID starting with 2abe449737d2a4866512f8b1892d9b76d018ecb616146a148ba2c9c2da897f77 not found: ID does not exist" Nov 03 22:37:22.332943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:22.332917 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:22.413838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:22.413808 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee489429-b3e4-4df7-ab1c-0ff2552cd802-kube-api-access-v8qht\" (UniqueName: \"kubernetes.io/projected/ee489429-b3e4-4df7-ab1c-0ff2552cd802-kube-api-access-v8qht\") pod \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " Nov 03 22:37:22.413838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:22.413847 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-openshift-ca\") pod \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " Nov 03 22:37:22.414124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:22.414067 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-xks-ca\") pod \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\" (UID: \"ee489429-b3e4-4df7-ab1c-0ff2552cd802\") " Nov 03 22:37:22.414314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:22.414281 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-openshift-ca" pod "ee489429-b3e4-4df7-ab1c-0ff2552cd802" (UID: "ee489429-b3e4-4df7-ab1c-0ff2552cd802"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:22.414760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:22.414740 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:22.416542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:22.416518 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-xks-ca" pod "ee489429-b3e4-4df7-ab1c-0ff2552cd802" (UID: "ee489429-b3e4-4df7-ab1c-0ff2552cd802"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:22.416628 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:22.416545 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee489429-b3e4-4df7-ab1c-0ff2552cd802-kube-api-access-v8qht" pod "ee489429-b3e4-4df7-ab1c-0ff2552cd802" (UID: "ee489429-b3e4-4df7-ab1c-0ff2552cd802"). InnerVolumeSpecName "kube-api-access-v8qht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:22.515894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:22.515855 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee489429-b3e4-4df7-ab1c-0ff2552cd802-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:22.515894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:22.515891 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8qht\" (UniqueName: \"kubernetes.io/projected/ee489429-b3e4-4df7-ab1c-0ff2552cd802-kube-api-access-v8qht\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:23.184791 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:23.184761 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" Nov 03 22:37:23.184969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:23.184789 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz" event={"ID":"ee489429-b3e4-4df7-ab1c-0ff2552cd802","Type":"ContainerDied","Data":"52d826d46cb7e554881e301c4929b738a6097657027ddc65af3f50a86f19296a"} Nov 03 22:37:23.184969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:23.184842 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52d826d46cb7e554881e301c4929b738a6097657027ddc65af3f50a86f19296a" Nov 03 22:37:24.765042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:24.764997 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:25.273435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:25.273403 4298 scope.go:122] "RemoveContainer" containerID="22ca6be03fe88115e499cdbaa0e4b7f97d29a8e6018599c3c880fafb74608300" Nov 03 22:37:25.283416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:25.283395 4298 scope.go:122] "RemoveContainer" containerID="f112f0f34be574566f5682a16278cd0ef0c34209ad15283e9da66d86b71dd710" Nov 03 22:37:28.851767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:28.851714 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:29.043859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:29.043827 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:29.371448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:29.371403 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:30.685966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:30.685934 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:30.865229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:30.865196 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:34.003055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:34.003016 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:38.458365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:38.458275 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:39.802664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:39.802603 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:40.106047 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:40.105960 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:43.271184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:43.271122 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:43.917086 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:43.917058 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:44.623353 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:44.623320 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:45.191260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.191211 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll"] Nov 03 22:37:45.193532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.193503 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ee489429-b3e4-4df7-ab1c-0ff2552cd802" containerName="revoke-keys" Nov 03 22:37:45.193740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.193723 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee489429-b3e4-4df7-ab1c-0ff2552cd802" containerName="revoke-keys" Nov 03 22:37:45.194081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.194067 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee489429-b3e4-4df7-ab1c-0ff2552cd802" containerName="revoke-keys" Nov 03 22:37:45.202552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.201384 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.236321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.236228 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:45.257214 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.254434 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a9ef23-openshift-default-dockercfg-w7x8n\"" Nov 03 22:37:45.267132 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.267100 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll"] Nov 03 22:37:45.383664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.367547 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-istio-token\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.383664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.367627 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvll\" (UniqueName: \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-kube-api-access-bmvll\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.383664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.367717 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-envoy\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.383664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.367805 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-certs\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.383664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.367863 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-socket\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.383664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.367886 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8131b38-abea-4933-91b4-e217facd68d8-istio-podinfo\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.383664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.367912 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8131b38-abea-4933-91b4-e217facd68d8-istiod-ca-cert\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.383664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.367997 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-credential-socket\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.383664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.368018 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-data\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.469366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.469330 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-envoy\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.469595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.469423 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-certs\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.469595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.469477 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-socket\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.469595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.469494 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8131b38-abea-4933-91b4-e217facd68d8-istio-podinfo\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.469595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.469519 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8131b38-abea-4933-91b4-e217facd68d8-istiod-ca-cert\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.469900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.469601 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-credential-socket\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.469900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.469626 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-data\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.469900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.469760 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-istio-token\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.469900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.469819 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvll\" (UniqueName: \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-kube-api-access-bmvll\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.470121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.469909 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-certs\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.470121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.469940 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-socket\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.470238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.470150 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-data\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.470238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.470200 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-credential-socket\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.470574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.470550 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8131b38-abea-4933-91b4-e217facd68d8-istiod-ca-cert\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.471849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.471827 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-envoy\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.472387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.472359 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8131b38-abea-4933-91b4-e217facd68d8-istio-podinfo\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.527734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.527695 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvll\" (UniqueName: \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-kube-api-access-bmvll\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.527931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.527907 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-istio-token\") pod \"e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.567979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.567945 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:45.762732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:45.762607 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:46.009403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:46.009368 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll"] Nov 03 22:37:46.041932 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:37:46.041904 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8131b38_abea_4933_91b4_e217facd68d8.slice/crio-d03c2b473f205eb24c47767e59546b0a62e8d98be76eb434c3a16194a9b66aa8 WatchSource:0}: Error finding container d03c2b473f205eb24c47767e59546b0a62e8d98be76eb434c3a16194a9b66aa8: Status 404 returned error can't find the container with id d03c2b473f205eb24c47767e59546b0a62e8d98be76eb434c3a16194a9b66aa8 Nov 03 22:37:46.044529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:46.044500 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:37:46.044610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:46.044568 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:37:46.044610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:46.044598 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:37:46.362304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:46.362206 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" event={"ID":"d8131b38-abea-4933-91b4-e217facd68d8","Type":"ContainerStarted","Data":"4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74"} Nov 03 22:37:46.362304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:46.362277 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" event={"ID":"d8131b38-abea-4933-91b4-e217facd68d8","Type":"ContainerStarted","Data":"d03c2b473f205eb24c47767e59546b0a62e8d98be76eb434c3a16194a9b66aa8"} Nov 03 22:37:46.481480 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:46.481414 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" podStartSLOduration=1.481394131 podStartE2EDuration="1.481394131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:46.478932984 +0000 UTC m=+1823.459666660" watchObservedRunningTime="2026-11-03 22:37:46.481394131 +0000 UTC m=+1823.462127785" Nov 03 22:37:46.568603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:46.568562 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:47.551592 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:37:47.551551 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:37:47.575448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:47.575420 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:48.290697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:48.290654 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:48.378589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:48.378554 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:48.379521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:48.379496 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:37:49.236003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:49.235972 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:49.485618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:49.485585 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:49.682900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:49.682815 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:53.484487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:53.484450 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:53.698319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:53.698273 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:54.189276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:54.189238 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:57.207865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.207831 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:57.389831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.389797 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:57.753628 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.753586 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4"] Nov 03 22:37:57.775672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.775446 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:37:57.802182 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.802151 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4"] Nov 03 22:37:57.843581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.843548 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a9ef23\"" Nov 03 22:37:57.843825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.843709 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a9ef23-dockercfg-r9vnt\"" Nov 03 22:37:57.864760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.864725 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59bce71a-9625-41ee-91f7-f361e31eb9c0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4\" (UID: \"59bce71a-9625-41ee-91f7-f361e31eb9c0\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:37:57.864973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.864862 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctlvq\" (UniqueName: \"kubernetes.io/projected/59bce71a-9625-41ee-91f7-f361e31eb9c0-kube-api-access-ctlvq\") pod \"payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4\" (UID: \"59bce71a-9625-41ee-91f7-f361e31eb9c0\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:37:57.913936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.913899 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v"] Nov 03 22:37:57.918279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.918256 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:37:57.966056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.966014 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v\" (UID: \"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:37:57.966251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.966227 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59bce71a-9625-41ee-91f7-f361e31eb9c0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4\" (UID: \"59bce71a-9625-41ee-91f7-f361e31eb9c0\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:37:57.966316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.966301 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzrm4\" (UniqueName: \"kubernetes.io/projected/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-kube-api-access-rzrm4\") pod \"payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v\" (UID: \"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:37:57.966376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.966355 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctlvq\" (UniqueName: \"kubernetes.io/projected/59bce71a-9625-41ee-91f7-f361e31eb9c0-kube-api-access-ctlvq\") pod \"payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4\" (UID: \"59bce71a-9625-41ee-91f7-f361e31eb9c0\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:37:57.967014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.966989 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59bce71a-9625-41ee-91f7-f361e31eb9c0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4\" (UID: \"59bce71a-9625-41ee-91f7-f361e31eb9c0\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:37:57.977231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:57.977194 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v"] Nov 03 22:37:58.024121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:58.024050 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctlvq\" (UniqueName: \"kubernetes.io/projected/59bce71a-9625-41ee-91f7-f361e31eb9c0-kube-api-access-ctlvq\") pod \"payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4\" (UID: \"59bce71a-9625-41ee-91f7-f361e31eb9c0\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:37:58.067927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:58.067885 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzrm4\" (UniqueName: \"kubernetes.io/projected/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-kube-api-access-rzrm4\") pod \"payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v\" (UID: \"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:37:58.068140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:58.068015 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v\" (UID: \"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:37:58.068828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:58.068800 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v\" (UID: \"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:37:58.099244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:58.099200 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:37:58.125953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:58.125925 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzrm4\" (UniqueName: \"kubernetes.io/projected/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-kube-api-access-rzrm4\") pod \"payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v\" (UID: \"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:37:58.239668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:58.238750 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:37:58.332725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:58.332351 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:58.481265 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:37:58.481232 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bce71a_9625_41ee_91f7_f361e31eb9c0.slice/crio-22c8c1002dbc01a97ace36749aabc6d945531b1b48531a851968d52364c123cd WatchSource:0}: Error finding container 22c8c1002dbc01a97ace36749aabc6d945531b1b48531a851968d52364c123cd: Status 404 returned error can't find the container with id 22c8c1002dbc01a97ace36749aabc6d945531b1b48531a851968d52364c123cd Nov 03 22:37:58.485211 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:58.485183 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4"] Nov 03 22:37:58.649100 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:37:58.649054 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9ffd2c_3fe3_42fc_a0cf_a7486b8a148c.slice/crio-78163ef30f4ebe2b53f88f25a82cc1d6e8ed487b0b716f596446270b93038364 WatchSource:0}: Error finding container 78163ef30f4ebe2b53f88f25a82cc1d6e8ed487b0b716f596446270b93038364: Status 404 returned error can't find the container with id 78163ef30f4ebe2b53f88f25a82cc1d6e8ed487b0b716f596446270b93038364 Nov 03 22:37:58.655276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:58.655244 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v"] Nov 03 22:37:58.796042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:58.796006 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:59.031187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.031155 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:59.112206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.112169 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h"] Nov 03 22:37:59.116566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.116544 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.171909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.171881 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-094053-openshift-default-dockercfg-nsksx\"" Nov 03 22:37:59.190243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.189579 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-certs\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.190534 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.190507 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gztx6\" (UniqueName: \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-kube-api-access-gztx6\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.190691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.190672 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-credential-socket\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.190851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.190825 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istiod-ca-cert\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.191036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.191018 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-socket\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.191374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.191333 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-token\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.191544 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.191527 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-data\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.191761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.191743 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-envoy\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.191909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.191893 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-podinfo\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.227346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.227314 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h"] Nov 03 22:37:59.252269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.252240 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:59.293501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.293399 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-token\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.293501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.293485 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-data\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.293776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.293552 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-envoy\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.293776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.293590 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-podinfo\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.293776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.293636 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-certs\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.293776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.293696 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gztx6\" (UniqueName: \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-kube-api-access-gztx6\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.293776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.293735 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-credential-socket\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.293776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.293765 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istiod-ca-cert\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.294109 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.293831 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-socket\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.296120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.294454 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-socket\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.296120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.295018 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-data\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.297728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.297109 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-credential-socket\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.297859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.297731 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istiod-ca-cert\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.297997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.297963 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-certs\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.302735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.302686 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-podinfo\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.303098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.303076 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-envoy\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.351940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.351910 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-token\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.352110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.351993 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gztx6\" (UniqueName: \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-kube-api-access-gztx6\") pod \"e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.430598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.430566 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:59.432697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.432674 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:37:59.468303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.468259 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" event={"ID":"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c","Type":"ContainerStarted","Data":"410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b"} Nov 03 22:37:59.468495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.468344 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" event={"ID":"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c","Type":"ContainerStarted","Data":"78163ef30f4ebe2b53f88f25a82cc1d6e8ed487b0b716f596446270b93038364"} Nov 03 22:37:59.468578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.468519 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:37:59.470855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.470823 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" event={"ID":"59bce71a-9625-41ee-91f7-f361e31eb9c0","Type":"ContainerStarted","Data":"64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029"} Nov 03 22:37:59.470999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.470874 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" event={"ID":"59bce71a-9625-41ee-91f7-f361e31eb9c0","Type":"ContainerStarted","Data":"22c8c1002dbc01a97ace36749aabc6d945531b1b48531a851968d52364c123cd"} Nov 03 22:37:59.470999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.470964 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:37:59.584719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.583513 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" podStartSLOduration=2.58349286 podStartE2EDuration="2.58349286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:59.582112271 +0000 UTC m=+1836.562845924" watchObservedRunningTime="2026-11-03 22:37:59.58349286 +0000 UTC m=+1836.564226514" Nov 03 22:37:59.705747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.704124 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" podStartSLOduration=2.704091088 podStartE2EDuration="2.704091088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:59.698074559 +0000 UTC m=+1836.678808243" watchObservedRunningTime="2026-11-03 22:37:59.704091088 +0000 UTC m=+1836.684824742" Nov 03 22:37:59.899722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.898803 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h"] Nov 03 22:37:59.907672 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:37:59.906398 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cf1bc3_4867_4446_866b_249d8cb2c7c3.slice/crio-75e2a5199f691d83baf9a29ef7539231a8225bbb8c4b9c57fe1752d89defbcbb WatchSource:0}: Error finding container 75e2a5199f691d83baf9a29ef7539231a8225bbb8c4b9c57fe1752d89defbcbb: Status 404 returned error can't find the container with id 75e2a5199f691d83baf9a29ef7539231a8225bbb8c4b9c57fe1752d89defbcbb Nov 03 22:37:59.917519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.911459 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:37:59.917519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.911553 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:37:59.917519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:37:59.911591 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:38:00.089684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.089623 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:00.376412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.376370 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w"] Nov 03 22:38:00.381076 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.381055 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.439676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.438952 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-e7d36a-openshift-default-dockercfg-cprdc\"" Nov 03 22:38:00.455326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.453655 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w"] Nov 03 22:38:00.494393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.494346 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" event={"ID":"68cf1bc3-4867-4446-866b-249d8cb2c7c3","Type":"ContainerStarted","Data":"0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef"} Nov 03 22:38:00.494599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.494410 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" event={"ID":"68cf1bc3-4867-4446-866b-249d8cb2c7c3","Type":"ContainerStarted","Data":"75e2a5199f691d83baf9a29ef7539231a8225bbb8c4b9c57fe1752d89defbcbb"} Nov 03 22:38:00.521543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.521507 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:00.521759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.521561 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-certs\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.522140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.522111 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-socket\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.522262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.522182 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-data\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.531439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.522943 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da18026-d752-43f1-a018-99c54759fb56-istio-podinfo\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.531439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.523042 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-credential-socket\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.531439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.523203 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-istio-token\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.531439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.523239 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-envoy\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.531439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.523275 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da18026-d752-43f1-a018-99c54759fb56-istiod-ca-cert\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.531439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.523315 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvk6x\" (UniqueName: \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-kube-api-access-kvk6x\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.643775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.642625 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-socket\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.644055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.644032 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-data\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.644258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.644242 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da18026-d752-43f1-a018-99c54759fb56-istio-podinfo\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.644443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.644424 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-credential-socket\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.644776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.644755 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-istio-token\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.644923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.644907 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-envoy\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.645058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.645043 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da18026-d752-43f1-a018-99c54759fb56-istiod-ca-cert\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.645278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.645222 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvk6x\" (UniqueName: \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-kube-api-access-kvk6x\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.645580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.645560 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-certs\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.646230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.646180 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-certs\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.646384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.643684 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-socket\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.647098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.647077 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-data\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.648503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.648197 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da18026-d752-43f1-a018-99c54759fb56-istiod-ca-cert\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.654430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.649490 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-credential-socket\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.677015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.676977 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-envoy\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.677743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.677717 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da18026-d752-43f1-a018-99c54759fb56-istio-podinfo\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.735576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.735541 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-istio-token\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.746732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.736346 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvk6x\" (UniqueName: \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-kube-api-access-kvk6x\") pod \"e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:00.778486 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:00.778452 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:01.000874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.000834 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:01.433671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.433561 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:38:01.441867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.441840 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:38:01.443829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.443785 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" podStartSLOduration=2.443770347 podStartE2EDuration="2.443770347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:00.654885715 +0000 UTC m=+1837.635619409" watchObservedRunningTime="2026-11-03 22:38:01.443770347 +0000 UTC m=+1838.424504006" Nov 03 22:38:01.445078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.445054 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w"] Nov 03 22:38:01.447358 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.447321 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:38:01.447469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.447404 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:38:01.447530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.447465 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:38:01.507170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.507137 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" event={"ID":"3da18026-d752-43f1-a018-99c54759fb56","Type":"ContainerStarted","Data":"8ebe27575702e6e3e373f4a6a96c377d8b91bff524182b15cdc7b2887f930497"} Nov 03 22:38:01.507373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.507352 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:38:01.508406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.508390 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:38:01.526890 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:01.526855 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:02.516009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:02.515971 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" event={"ID":"3da18026-d752-43f1-a018-99c54759fb56","Type":"ContainerStarted","Data":"58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb"} Nov 03 22:38:02.687809 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:02.687739 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" podStartSLOduration=2.687724676 podStartE2EDuration="2.687724676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:02.683072207 +0000 UTC m=+1839.663805861" watchObservedRunningTime="2026-11-03 22:38:02.687724676 +0000 UTC m=+1839.668458328" Nov 03 22:38:03.001723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:03.001679 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:03.008508 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:03.008476 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:03.523530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:03.523476 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:03.524576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:03.524551 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:38:03.647598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:03.647567 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:03.831387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:03.831291 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:04.344821 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:04.344670 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:06.355519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:06.355486 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:06.561606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:06.561574 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:07.847635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:07.847604 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:09.303961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:09.303928 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:09.529999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:09.529960 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:09.715501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:09.715471 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:10.221786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.221752 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:10.408662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.407156 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:10.520555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.520470 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:38:10.523332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.523308 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:38:10.742854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.742813 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z"] Nov 03 22:38:10.748551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.748518 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:38:10.814157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.813990 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-094053\"" Nov 03 22:38:10.814692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.814675 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-094053-dockercfg-6htxr\"" Nov 03 22:38:10.826674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.826602 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z\" (UID: \"3541be3e-2e43-4741-a8ac-a3cd3d5df25d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:38:10.833564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.833535 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxszs\" (UniqueName: \"kubernetes.io/projected/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-kube-api-access-zxszs\") pod \"payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z\" (UID: \"3541be3e-2e43-4741-a8ac-a3cd3d5df25d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:38:10.862489 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.862457 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z"] Nov 03 22:38:10.928287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.928248 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x"] Nov 03 22:38:10.933611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.933581 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:38:10.934805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.934777 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z\" (UID: \"3541be3e-2e43-4741-a8ac-a3cd3d5df25d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:38:10.934899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.934841 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxszs\" (UniqueName: \"kubernetes.io/projected/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-kube-api-access-zxszs\") pod \"payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z\" (UID: \"3541be3e-2e43-4741-a8ac-a3cd3d5df25d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:38:10.935431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.935409 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z\" (UID: \"3541be3e-2e43-4741-a8ac-a3cd3d5df25d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:38:10.990075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.990036 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x"] Nov 03 22:38:10.994729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:10.994702 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxszs\" (UniqueName: \"kubernetes.io/projected/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-kube-api-access-zxszs\") pod \"payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z\" (UID: \"3541be3e-2e43-4741-a8ac-a3cd3d5df25d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:38:11.037072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:11.037022 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd9ab68-16fa-4867-9a8d-11b880e416ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x\" (UID: \"2cd9ab68-16fa-4867-9a8d-11b880e416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:38:11.037266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:11.037198 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km6v8\" (UniqueName: \"kubernetes.io/projected/2cd9ab68-16fa-4867-9a8d-11b880e416ca-kube-api-access-km6v8\") pod \"payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x\" (UID: \"2cd9ab68-16fa-4867-9a8d-11b880e416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:38:11.138457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:11.138370 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km6v8\" (UniqueName: \"kubernetes.io/projected/2cd9ab68-16fa-4867-9a8d-11b880e416ca-kube-api-access-km6v8\") pod \"payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x\" (UID: \"2cd9ab68-16fa-4867-9a8d-11b880e416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:38:11.138631 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:11.138522 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd9ab68-16fa-4867-9a8d-11b880e416ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x\" (UID: \"2cd9ab68-16fa-4867-9a8d-11b880e416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:38:11.139203 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:11.139181 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd9ab68-16fa-4867-9a8d-11b880e416ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x\" (UID: \"2cd9ab68-16fa-4867-9a8d-11b880e416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:38:11.167218 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:11.167182 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:38:11.196060 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:11.196025 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km6v8\" (UniqueName: \"kubernetes.io/projected/2cd9ab68-16fa-4867-9a8d-11b880e416ca-kube-api-access-km6v8\") pod \"payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x\" (UID: \"2cd9ab68-16fa-4867-9a8d-11b880e416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:38:11.252264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:11.251454 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:38:11.559877 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:38:11.559844 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3541be3e_2e43_4741_a8ac_a3cd3d5df25d.slice/crio-cf6b59fdf26260a4030612fee9bbb71d367370ebb9693ae1363f5d2ead1d4607 WatchSource:0}: Error finding container cf6b59fdf26260a4030612fee9bbb71d367370ebb9693ae1363f5d2ead1d4607: Status 404 returned error can't find the container with id cf6b59fdf26260a4030612fee9bbb71d367370ebb9693ae1363f5d2ead1d4607 Nov 03 22:38:11.565042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:11.564997 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z"] Nov 03 22:38:11.589263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:11.589231 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" event={"ID":"3541be3e-2e43-4741-a8ac-a3cd3d5df25d","Type":"ContainerStarted","Data":"cf6b59fdf26260a4030612fee9bbb71d367370ebb9693ae1363f5d2ead1d4607"} Nov 03 22:38:11.680824 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:38:11.680788 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd9ab68_16fa_4867_9a8d_11b880e416ca.slice/crio-79c3970ab518fc74ec585e5c68156e67830a5ea889ce88d2dad4dec42cc8efdf WatchSource:0}: Error finding container 79c3970ab518fc74ec585e5c68156e67830a5ea889ce88d2dad4dec42cc8efdf: Status 404 returned error can't find the container with id 79c3970ab518fc74ec585e5c68156e67830a5ea889ce88d2dad4dec42cc8efdf Nov 03 22:38:11.684505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:11.684480 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x"] Nov 03 22:38:12.597749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:12.597711 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" event={"ID":"3541be3e-2e43-4741-a8ac-a3cd3d5df25d","Type":"ContainerStarted","Data":"e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72"} Nov 03 22:38:12.598242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:12.597792 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:38:12.599369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:12.599347 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" event={"ID":"2cd9ab68-16fa-4867-9a8d-11b880e416ca","Type":"ContainerStarted","Data":"88e7f514ee9b641f4ae87153947ba933ddd8320aac455fcade115a7df434e3b6"} Nov 03 22:38:12.599473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:12.599390 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:38:12.599473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:12.599406 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" event={"ID":"2cd9ab68-16fa-4867-9a8d-11b880e416ca","Type":"ContainerStarted","Data":"79c3970ab518fc74ec585e5c68156e67830a5ea889ce88d2dad4dec42cc8efdf"} Nov 03 22:38:12.712838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:12.712786 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" podStartSLOduration=2.712773062 podStartE2EDuration="2.712773062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:12.710773844 +0000 UTC m=+1849.691507497" watchObservedRunningTime="2026-11-03 22:38:12.712773062 +0000 UTC m=+1849.693506741" Nov 03 22:38:12.827680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:12.827594 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" podStartSLOduration=2.827578662 podStartE2EDuration="2.827578662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:12.824377696 +0000 UTC m=+1849.805111352" watchObservedRunningTime="2026-11-03 22:38:12.827578662 +0000 UTC m=+1849.808312315" Nov 03 22:38:12.996922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:12.996889 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:13.173146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.173104 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:13.379188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.379107 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq"] Nov 03 22:38:13.383917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.383892 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:38:13.436774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.436743 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e7d36a\"" Nov 03 22:38:13.439465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.439439 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq"] Nov 03 22:38:13.478946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.478904 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xnkp\" (UniqueName: \"kubernetes.io/projected/c5bc5843-6681-4320-ae7f-0b8eda6e2961-kube-api-access-5xnkp\") pod \"payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq\" (UID: \"c5bc5843-6681-4320-ae7f-0b8eda6e2961\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:38:13.479249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.479221 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bc5843-6681-4320-ae7f-0b8eda6e2961-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq\" (UID: \"c5bc5843-6681-4320-ae7f-0b8eda6e2961\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:38:13.530533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.530488 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7"] Nov 03 22:38:13.535533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.535509 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:38:13.581138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.581079 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bc5843-6681-4320-ae7f-0b8eda6e2961-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq\" (UID: \"c5bc5843-6681-4320-ae7f-0b8eda6e2961\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:38:13.581383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.581359 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xnkp\" (UniqueName: \"kubernetes.io/projected/c5bc5843-6681-4320-ae7f-0b8eda6e2961-kube-api-access-5xnkp\") pod \"payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq\" (UID: \"c5bc5843-6681-4320-ae7f-0b8eda6e2961\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:38:13.582106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.582025 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bc5843-6681-4320-ae7f-0b8eda6e2961-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq\" (UID: \"c5bc5843-6681-4320-ae7f-0b8eda6e2961\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:38:13.591368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.591324 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7"] Nov 03 22:38:13.591833 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.591806 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e7d36a-dockercfg-7tpv2\"" Nov 03 22:38:13.639607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.639517 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xnkp\" (UniqueName: \"kubernetes.io/projected/c5bc5843-6681-4320-ae7f-0b8eda6e2961-kube-api-access-5xnkp\") pod \"payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq\" (UID: \"c5bc5843-6681-4320-ae7f-0b8eda6e2961\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:38:13.673597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.673559 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:13.683307 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.683270 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f607014-d55b-45d4-943a-f1a662f9edfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7\" (UID: \"8f607014-d55b-45d4-943a-f1a662f9edfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:38:13.683483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.683395 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgvw6\" (UniqueName: \"kubernetes.io/projected/8f607014-d55b-45d4-943a-f1a662f9edfd-kube-api-access-kgvw6\") pod \"payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7\" (UID: \"8f607014-d55b-45d4-943a-f1a662f9edfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:38:13.703936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.703892 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:38:13.785358 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.784935 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f607014-d55b-45d4-943a-f1a662f9edfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7\" (UID: \"8f607014-d55b-45d4-943a-f1a662f9edfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:38:13.785358 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.785049 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgvw6\" (UniqueName: \"kubernetes.io/projected/8f607014-d55b-45d4-943a-f1a662f9edfd-kube-api-access-kgvw6\") pod \"payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7\" (UID: \"8f607014-d55b-45d4-943a-f1a662f9edfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:38:13.786297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.786271 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f607014-d55b-45d4-943a-f1a662f9edfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7\" (UID: \"8f607014-d55b-45d4-943a-f1a662f9edfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:38:13.854560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.854529 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgvw6\" (UniqueName: \"kubernetes.io/projected/8f607014-d55b-45d4-943a-f1a662f9edfd-kube-api-access-kgvw6\") pod \"payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7\" (UID: \"8f607014-d55b-45d4-943a-f1a662f9edfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:38:13.860109 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:13.860063 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:38:14.093783 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:38:14.093746 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5bc5843_6681_4320_ae7f_0b8eda6e2961.slice/crio-75a893a6682b694b9115b7070808497823882455052302203b24bdd3a20b24cb WatchSource:0}: Error finding container 75a893a6682b694b9115b7070808497823882455052302203b24bdd3a20b24cb: Status 404 returned error can't find the container with id 75a893a6682b694b9115b7070808497823882455052302203b24bdd3a20b24cb Nov 03 22:38:14.102687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:14.098882 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq"] Nov 03 22:38:14.253428 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:38:14.253394 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f607014_d55b_45d4_943a_f1a662f9edfd.slice/crio-82c2338bf9bf32350943135dc78fac2eb7f14ecb8c4134c2e44835aa7a846a31 WatchSource:0}: Error finding container 82c2338bf9bf32350943135dc78fac2eb7f14ecb8c4134c2e44835aa7a846a31: Status 404 returned error can't find the container with id 82c2338bf9bf32350943135dc78fac2eb7f14ecb8c4134c2e44835aa7a846a31 Nov 03 22:38:14.258111 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:14.258088 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7"] Nov 03 22:38:14.616210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:14.616167 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" event={"ID":"c5bc5843-6681-4320-ae7f-0b8eda6e2961","Type":"ContainerStarted","Data":"25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937"} Nov 03 22:38:14.616397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:14.616232 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" event={"ID":"c5bc5843-6681-4320-ae7f-0b8eda6e2961","Type":"ContainerStarted","Data":"75a893a6682b694b9115b7070808497823882455052302203b24bdd3a20b24cb"} Nov 03 22:38:14.616397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:14.616260 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:38:14.617900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:14.617875 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" event={"ID":"8f607014-d55b-45d4-943a-f1a662f9edfd","Type":"ContainerStarted","Data":"ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe"} Nov 03 22:38:14.617996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:14.617921 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" event={"ID":"8f607014-d55b-45d4-943a-f1a662f9edfd","Type":"ContainerStarted","Data":"82c2338bf9bf32350943135dc78fac2eb7f14ecb8c4134c2e44835aa7a846a31"} Nov 03 22:38:14.617996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:14.617943 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:38:14.732756 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:14.732706 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" podStartSLOduration=1.73268831 podStartE2EDuration="1.73268831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:14.729706942 +0000 UTC m=+1851.710440596" watchObservedRunningTime="2026-11-03 22:38:14.73268831 +0000 UTC m=+1851.713421962" Nov 03 22:38:14.861154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:14.861087 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" podStartSLOduration=1.861059332 podStartE2EDuration="1.861059332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:14.856871483 +0000 UTC m=+1851.837605136" watchObservedRunningTime="2026-11-03 22:38:14.861059332 +0000 UTC m=+1851.841792987" Nov 03 22:38:15.796600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:15.796560 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:17.553718 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:38:17.553677 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:38:20.306139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:20.306107 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:23.611554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:23.611523 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:38:23.612046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:23.611600 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:38:24.807661 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:24.807336 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:25.627725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:25.627696 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:38:25.627934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:25.627777 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:38:26.049033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:26.048995 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j"] Nov 03 22:38:26.053910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:26.053886 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248efb3a-7ss4j"] Nov 03 22:38:27.140708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:27.140668 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d106648c-97a0-41e5-8c03-3d788bca08cb" path="/var/lib/kubelet/pods/d106648c-97a0-41e5-8c03-3d788bca08cb/volumes" Nov 03 22:38:29.286434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:29.286372 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:34.394479 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:34.394443 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:38.277698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.277597 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm"] Nov 03 22:38:38.283279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.283260 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:38.380193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.380164 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:38:38.380551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.380512 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:38.384541 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.384511 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm"] Nov 03 22:38:38.404137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.404104 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48738db1-5pljm\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:38.404437 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.404405 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4rwz\" (UniqueName: \"kubernetes.io/projected/40c92f85-e307-4ca6-8e7e-6089d34b3a50-kube-api-access-d4rwz\") pod \"maas-api-revoke-sub-48738db1-5pljm\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:38.404562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.404459 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48738db1-5pljm\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:38.506038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.506004 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48738db1-5pljm\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:38.506217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.506094 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4rwz\" (UniqueName: \"kubernetes.io/projected/40c92f85-e307-4ca6-8e7e-6089d34b3a50-kube-api-access-d4rwz\") pod \"maas-api-revoke-sub-48738db1-5pljm\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:38.506217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.506140 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48738db1-5pljm\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:38.506738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.506718 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48738db1-5pljm\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:38.508745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.508728 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48738db1-5pljm\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:38.574527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.574442 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4rwz\" (UniqueName: \"kubernetes.io/projected/40c92f85-e307-4ca6-8e7e-6089d34b3a50-kube-api-access-d4rwz\") pod \"maas-api-revoke-sub-48738db1-5pljm\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:38.598007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:38.597979 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:39.016812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:39.016780 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:39.093602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:39.093545 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm"] Nov 03 22:38:39.881424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:39.881383 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" event={"ID":"40c92f85-e307-4ca6-8e7e-6089d34b3a50","Type":"ContainerStarted","Data":"0644a1f61a8f5ce40e09f20354addac250027fbb01359604f38f1230e04cda24"} Nov 03 22:38:39.881974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:39.881471 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" event={"ID":"40c92f85-e307-4ca6-8e7e-6089d34b3a50","Type":"ContainerStarted","Data":"649f2fdd4c395f3ae9c7f0f558966f3b61910317c875ae39dbb94dda8acf530e"} Nov 03 22:38:40.003375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:40.003310 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" podStartSLOduration=2.003289614 podStartE2EDuration="2.003289614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:39.994520854 +0000 UTC m=+1876.975254507" watchObservedRunningTime="2026-11-03 22:38:40.003289614 +0000 UTC m=+1876.984023270" Nov 03 22:38:40.490844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:40.490810 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g"] Nov 03 22:38:40.491064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:40.491036 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" containerName="istio-proxy" containerID="cri-o://c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec" gracePeriod=30 Nov 03 22:38:40.893819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:40.893778 4298 generic.go:356] "Generic (PLEG): container finished" podID="40c92f85-e307-4ca6-8e7e-6089d34b3a50" containerID="0644a1f61a8f5ce40e09f20354addac250027fbb01359604f38f1230e04cda24" exitCode=0 Nov 03 22:38:40.894386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:40.894045 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" event={"ID":"40c92f85-e307-4ca6-8e7e-6089d34b3a50","Type":"ContainerDied","Data":"0644a1f61a8f5ce40e09f20354addac250027fbb01359604f38f1230e04cda24"} Nov 03 22:38:42.082432 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.082405 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:42.162175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.162092 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c92f85-e307-4ca6-8e7e-6089d34b3a50-kube-api-access-d4rwz\" (UniqueName: \"kubernetes.io/projected/40c92f85-e307-4ca6-8e7e-6089d34b3a50-kube-api-access-d4rwz\") pod \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " Nov 03 22:38:42.162175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.162172 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-xks-ca\") pod \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " Nov 03 22:38:42.162407 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.162312 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-openshift-ca\") pod \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\" (UID: \"40c92f85-e307-4ca6-8e7e-6089d34b3a50\") " Nov 03 22:38:42.162707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.162673 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-openshift-ca" pod "40c92f85-e307-4ca6-8e7e-6089d34b3a50" (UID: "40c92f85-e307-4ca6-8e7e-6089d34b3a50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:42.163131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.163105 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:42.164414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.164374 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-xks-ca" pod "40c92f85-e307-4ca6-8e7e-6089d34b3a50" (UID: "40c92f85-e307-4ca6-8e7e-6089d34b3a50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:42.164586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.164559 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c92f85-e307-4ca6-8e7e-6089d34b3a50-kube-api-access-d4rwz" pod "40c92f85-e307-4ca6-8e7e-6089d34b3a50" (UID: "40c92f85-e307-4ca6-8e7e-6089d34b3a50"). InnerVolumeSpecName "kube-api-access-d4rwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:42.264575 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.264534 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4rwz\" (UniqueName: \"kubernetes.io/projected/40c92f85-e307-4ca6-8e7e-6089d34b3a50-kube-api-access-d4rwz\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:42.264575 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.264578 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c92f85-e307-4ca6-8e7e-6089d34b3a50-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:42.913012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.912975 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" event={"ID":"40c92f85-e307-4ca6-8e7e-6089d34b3a50","Type":"ContainerDied","Data":"649f2fdd4c395f3ae9c7f0f558966f3b61910317c875ae39dbb94dda8acf530e"} Nov 03 22:38:42.913207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.913033 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="649f2fdd4c395f3ae9c7f0f558966f3b61910317c875ae39dbb94dda8acf530e" Nov 03 22:38:42.913207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:42.912988 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm" Nov 03 22:38:43.499950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.492944 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:43.922032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.921948 4298 generic.go:356] "Generic (PLEG): container finished" podID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerID="f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee" exitCode=0 Nov 03 22:38:43.922032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.922020 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" event={"ID":"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0","Type":"ContainerDied","Data":"f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee"} Nov 03 22:38:43.922516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.922491 4298 scope.go:122] "RemoveContainer" containerID="f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee" Nov 03 22:38:43.923979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.923949 4298 generic.go:356] "Generic (PLEG): container finished" podID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerID="abe257aca0704ed70808099dd8dedbeb68c9e10d08f0f74fc55f4d744f03a57c" exitCode=0 Nov 03 22:38:43.924081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.924023 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" event={"ID":"1ef2fc6c-a501-4a62-adcf-879a017e7bd8","Type":"ContainerDied","Data":"abe257aca0704ed70808099dd8dedbeb68c9e10d08f0f74fc55f4d744f03a57c"} Nov 03 22:38:43.924472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.924454 4298 scope.go:122] "RemoveContainer" containerID="abe257aca0704ed70808099dd8dedbeb68c9e10d08f0f74fc55f4d744f03a57c" Nov 03 22:38:43.924575 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.924547 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:38:43.924677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.924635 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:38:43.924769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.924705 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:38:43.926334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.926297 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:38:43.926430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.926378 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:38:43.926430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:43.926422 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:38:44.596738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:44.596696 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:38:44.597396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:44.596766 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:38:44.979203 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:44.979159 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" event={"ID":"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0","Type":"ContainerStarted","Data":"d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4"} Nov 03 22:38:44.982137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:44.982093 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" event={"ID":"1ef2fc6c-a501-4a62-adcf-879a017e7bd8","Type":"ContainerStarted","Data":"5c791e8d595cbe2021c6e1941f98f840af81fa9dbe30c0375b521fb3b2521dfc"} Nov 03 22:38:45.674668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.674352 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:38:45.676135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.675994 4298 patch_prober.go:29] interesting pod/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 03 22:38:45.676135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.676054 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 03 22:38:45.801980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.801789 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:38:45.802628 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.802600 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 22:38:45.802758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.802669 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 22:38:45.914194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.914161 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:38:45.991135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.991097 4298 generic.go:356] "Generic (PLEG): container finished" podID="189b8cc5-c678-4a68-81b9-144c292c18af" containerID="c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec" exitCode=0 Nov 03 22:38:45.991310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.991148 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" Nov 03 22:38:45.991310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.991207 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" event={"ID":"189b8cc5-c678-4a68-81b9-144c292c18af","Type":"ContainerDied","Data":"c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec"} Nov 03 22:38:45.991310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.991244 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g" event={"ID":"189b8cc5-c678-4a68-81b9-144c292c18af","Type":"ContainerDied","Data":"cd15fb10f45d370eb96242a7940128f81e6995bf7ad222831aa41c53ba8214e9"} Nov 03 22:38:45.991310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:45.991266 4298 scope.go:122] "RemoveContainer" containerID="c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec" Nov 03 22:38:46.002028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.002007 4298 scope.go:122] "RemoveContainer" containerID="c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec" Nov 03 22:38:46.002344 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:38:46.002314 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec\": container with ID starting with c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec not found: ID does not exist" containerID="c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec" Nov 03 22:38:46.002409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.002354 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec"} err="failed to get container status \"c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec\": rpc error: code = NotFound desc = could not find container \"c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec\": container with ID starting with c45aefa4fa9484f836eadc50a9bb21b5671e88f97c1189c81cc492195b7f03ec not found: ID does not exist" Nov 03 22:38:46.034676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.023925 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-data\") pod \"189b8cc5-c678-4a68-81b9-144c292c18af\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " Nov 03 22:38:46.034676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.024015 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-envoy\") pod \"189b8cc5-c678-4a68-81b9-144c292c18af\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " Nov 03 22:38:46.034676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.024100 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189b8cc5-c678-4a68-81b9-144c292c18af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189b8cc5-c678-4a68-81b9-144c292c18af-istiod-ca-cert\") pod \"189b8cc5-c678-4a68-81b9-144c292c18af\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " Nov 03 22:38:46.034676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.024180 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-kube-api-access-6wcl9\" (UniqueName: \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-kube-api-access-6wcl9\") pod \"189b8cc5-c678-4a68-81b9-144c292c18af\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " Nov 03 22:38:46.034676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.024210 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-socket\") pod \"189b8cc5-c678-4a68-81b9-144c292c18af\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " Nov 03 22:38:46.034676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.024271 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-credential-socket\") pod \"189b8cc5-c678-4a68-81b9-144c292c18af\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " Nov 03 22:38:46.034676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.024373 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/189b8cc5-c678-4a68-81b9-144c292c18af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189b8cc5-c678-4a68-81b9-144c292c18af-istio-podinfo\") pod \"189b8cc5-c678-4a68-81b9-144c292c18af\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " Nov 03 22:38:46.034676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.024450 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-istio-token\" (UniqueName: \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-istio-token\") pod \"189b8cc5-c678-4a68-81b9-144c292c18af\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " Nov 03 22:38:46.034676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.024513 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-certs\") pod \"189b8cc5-c678-4a68-81b9-144c292c18af\" (UID: \"189b8cc5-c678-4a68-81b9-144c292c18af\") " Nov 03 22:38:46.034676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.033991 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-data" pod "189b8cc5-c678-4a68-81b9-144c292c18af" (UID: "189b8cc5-c678-4a68-81b9-144c292c18af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:46.037675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.035449 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-certs" pod "189b8cc5-c678-4a68-81b9-144c292c18af" (UID: "189b8cc5-c678-4a68-81b9-144c292c18af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:46.040678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.037928 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-socket" pod "189b8cc5-c678-4a68-81b9-144c292c18af" (UID: "189b8cc5-c678-4a68-81b9-144c292c18af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:46.041352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.041013 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189b8cc5-c678-4a68-81b9-144c292c18af-istiod-ca-cert" pod "189b8cc5-c678-4a68-81b9-144c292c18af" (UID: "189b8cc5-c678-4a68-81b9-144c292c18af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:46.042408 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.042373 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-credential-socket" pod "189b8cc5-c678-4a68-81b9-144c292c18af" (UID: "189b8cc5-c678-4a68-81b9-144c292c18af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:46.079964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.078427 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/189b8cc5-c678-4a68-81b9-144c292c18af-istio-podinfo" pod "189b8cc5-c678-4a68-81b9-144c292c18af" (UID: "189b8cc5-c678-4a68-81b9-144c292c18af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:46.089784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.089730 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-envoy" pod "189b8cc5-c678-4a68-81b9-144c292c18af" (UID: "189b8cc5-c678-4a68-81b9-144c292c18af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:46.090546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.090514 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-istio-token" pod "189b8cc5-c678-4a68-81b9-144c292c18af" (UID: "189b8cc5-c678-4a68-81b9-144c292c18af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:46.091413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.091389 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-kube-api-access-6wcl9" pod "189b8cc5-c678-4a68-81b9-144c292c18af" (UID: "189b8cc5-c678-4a68-81b9-144c292c18af"). InnerVolumeSpecName "kube-api-access-6wcl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:46.177123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.175190 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189b8cc5-c678-4a68-81b9-144c292c18af-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.177123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.175231 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.177123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.175248 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.177123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.175262 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.177123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.175277 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.177123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.175291 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189b8cc5-c678-4a68-81b9-144c292c18af-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.177123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.175309 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wcl9\" (UniqueName: \"kubernetes.io/projected/189b8cc5-c678-4a68-81b9-144c292c18af-kube-api-access-6wcl9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.177123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.175324 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.177123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.175339 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189b8cc5-c678-4a68-81b9-144c292c18af-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.419892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.419807 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g"] Nov 03 22:38:46.450670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.447521 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-m724g"] Nov 03 22:38:46.673819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.670131 4298 patch_prober.go:29] interesting pod/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 03 22:38:46.673819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.670201 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 03 22:38:46.801936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.801902 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 22:38:46.802442 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:46.801959 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 22:38:47.197318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:47.197284 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" path="/var/lib/kubelet/pods/189b8cc5-c678-4a68-81b9-144c292c18af/volumes" Nov 03 22:38:47.562726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:47.559792 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:38:47.684670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:47.668052 4298 patch_prober.go:29] interesting pod/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 03 22:38:47.684670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:47.668113 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 03 22:38:47.696296 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:38:47.696259 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:38:47.801555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:47.801524 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 22:38:47.801743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:47.801576 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 22:38:48.310730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:48.310697 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:38:48.665191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:48.665085 4298 patch_prober.go:29] interesting pod/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 03 22:38:48.665482 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:48.665446 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 03 22:38:48.801656 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:48.801607 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 22:38:48.801865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:48.801681 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 22:38:49.672141 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:49.672098 4298 patch_prober.go:29] interesting pod/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 03 22:38:49.672797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:49.672754 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 03 22:38:49.802051 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:49.802013 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 22:38:49.802236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:49.802071 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 22:38:50.669911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:50.669881 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:38:50.671123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:50.671101 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:38:50.806381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:50.806349 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:38:50.807294 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:38:50.807270 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:39:10.056937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:10.056893 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc"] Nov 03 22:39:10.102599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:10.102566 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3174043-6j8xc"] Nov 03 22:39:11.136940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:11.136910 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab391ca-993d-40cf-bf77-b5ed2618951d" path="/var/lib/kubelet/pods/8ab391ca-993d-40cf-bf77-b5ed2618951d/volumes" Nov 03 22:39:14.024838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:14.024807 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:17.555791 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:39:17.555752 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:39:22.671595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:22.671558 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:25.362492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:25.362456 4298 scope.go:122] "RemoveContainer" containerID="f42de6f57666696665987127075b9441f025c8fc8d4424f47866a0e7d192f6f0" Nov 03 22:39:25.372398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:25.372366 4298 scope.go:122] "RemoveContainer" containerID="2f17f61fa2e565f50dede7dd20fc1c2666bf673f5a792b7e759517937d4cbc3a" Nov 03 22:39:25.381388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:25.381371 4298 scope.go:122] "RemoveContainer" containerID="eb8a3e9f55e18602dfdbb0b113990e4f08c09fb3f4165cc61220e6303183bf51" Nov 03 22:39:25.390343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:25.390322 4298 scope.go:122] "RemoveContainer" containerID="889a5753db6a4fb59e22fb67a520620f1f83680cf32bd589912604072e9eb0fd" Nov 03 22:39:25.400955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:25.400932 4298 scope.go:122] "RemoveContainer" containerID="1338a7530bd79a258ce3b0910fbb8453c7d2a36fffb5e80be5050b5dc44f7519" Nov 03 22:39:25.411653 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:25.411618 4298 scope.go:122] "RemoveContainer" containerID="cb055a8ada08f007de06c464106c65fc65811a65d699bdf0dcd9c21cae790af7" Nov 03 22:39:27.113152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:27.113122 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:31.589218 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:31.589088 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:31.958897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:31.958866 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:32.973878 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:32.973779 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:33.214046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:33.213955 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:33.403928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:33.403788 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:33.592085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:33.592038 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:36.316546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:36.316491 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:36.502306 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:36.502276 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:38.027658 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:38.027606 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:38.442550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:38.442466 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:38.620871 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:38.620824 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:38.896928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:38.896890 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:39.546850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:39.546817 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:39.736589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:39.736560 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:40.843037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:40.843006 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:42.798887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:42.798809 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:43.041161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:43.041125 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:43.298952 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:43.298912 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:43.497702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:43.497667 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz"] Nov 03 22:39:43.546705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:43.546672 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-gjgsz"] Nov 03 22:39:45.059854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:45.059816 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:45.138422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:45.138388 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee489429-b3e4-4df7-ab1c-0ff2552cd802" path="/var/lib/kubelet/pods/ee489429-b3e4-4df7-ab1c-0ff2552cd802/volumes" Nov 03 22:39:45.249080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:45.249040 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:46.956242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:46.956202 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:47.145322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:47.145243 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:47.338305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:47.338214 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:47.536486 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:47.536443 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:47.685214 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:39:47.683489 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:39:50.672298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:50.672265 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:50.873804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:50.873772 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:52.481851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:52.481813 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:54.258686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:54.258632 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:54.445568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:54.445535 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:57.468704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:57.468632 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:57.687165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:57.687070 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:57.932624 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:57.932512 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:58.371359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:58.371323 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:59.217564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:59.217534 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:59.397585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:39:59.397553 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:02.683330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:02.683297 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:07.416021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.415983 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk"] Nov 03 22:40:07.416670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.416628 4298 cpu_manager.go:414] "containerMap: removing container" podUID="40c92f85-e307-4ca6-8e7e-6089d34b3a50" containerName="revoke-keys" Nov 03 22:40:07.416670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.416658 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c92f85-e307-4ca6-8e7e-6089d34b3a50" containerName="revoke-keys" Nov 03 22:40:07.416769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.416719 4298 cpu_manager.go:414] "containerMap: removing container" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" containerName="istio-proxy" Nov 03 22:40:07.416769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.416725 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" containerName="istio-proxy" Nov 03 22:40:07.416873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.416863 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="189b8cc5-c678-4a68-81b9-144c292c18af" containerName="istio-proxy" Nov 03 22:40:07.416917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.416878 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c92f85-e307-4ca6-8e7e-6089d34b3a50" containerName="revoke-keys" Nov 03 22:40:07.420814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.420781 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:07.473576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.473541 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:40:07.485273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.485234 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:07.485440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.485295 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7tzf\" (UniqueName: \"kubernetes.io/projected/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-kube-api-access-h7tzf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:07.485500 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.485481 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:07.488743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.488718 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:07.575115 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.574169 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:40:07.584730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.584693 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk"] Nov 03 22:40:07.587226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.587185 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:07.587380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.587357 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:07.587444 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.587417 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7tzf\" (UniqueName: \"kubernetes.io/projected/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-kube-api-access-h7tzf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:07.589212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.589181 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:07.594426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.594394 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:07.652437 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.652401 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7tzf\" (UniqueName: \"kubernetes.io/projected/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-kube-api-access-h7tzf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:07.758541 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:07.758501 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:08.054686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:08.054566 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc"] Nov 03 22:40:08.059376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:08.059337 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed2f404-24jqc"] Nov 03 22:40:08.154038 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:40:08.154009 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27c49d4_2222_4f4b_9b4f_e1734b0ea57e.slice/crio-3a9245deb36b86d471bf12afd557945e3f7be8e844590a7b965b698fb5adecaa WatchSource:0}: Error finding container 3a9245deb36b86d471bf12afd557945e3f7be8e844590a7b965b698fb5adecaa: Status 404 returned error can't find the container with id 3a9245deb36b86d471bf12afd557945e3f7be8e844590a7b965b698fb5adecaa Nov 03 22:40:08.156775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:08.156752 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk"] Nov 03 22:40:08.974826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:08.974667 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" event={"ID":"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e","Type":"ContainerStarted","Data":"2c2333068efd42e99089ad5f06c6b78582738abced2e628720225affc778df95"} Nov 03 22:40:08.974826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:08.974739 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" event={"ID":"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e","Type":"ContainerStarted","Data":"3a9245deb36b86d471bf12afd557945e3f7be8e844590a7b965b698fb5adecaa"} Nov 03 22:40:09.094251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:09.094187 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" podStartSLOduration=2.094171893 podStartE2EDuration="2.094171893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:09.089852535 +0000 UTC m=+1966.070586193" watchObservedRunningTime="2026-11-03 22:40:09.094171893 +0000 UTC m=+1966.074905545" Nov 03 22:40:09.146235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:09.146197 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931311a4-60e0-4d4c-a60d-0714824c1933" path="/var/lib/kubelet/pods/931311a4-60e0-4d4c-a60d-0714824c1933/volumes" Nov 03 22:40:09.983249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:09.983218 4298 generic.go:356] "Generic (PLEG): container finished" podID="b27c49d4-2222-4f4b-9b4f-e1734b0ea57e" containerID="2c2333068efd42e99089ad5f06c6b78582738abced2e628720225affc778df95" exitCode=0 Nov 03 22:40:09.983655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:09.983273 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" event={"ID":"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e","Type":"ContainerDied","Data":"2c2333068efd42e99089ad5f06c6b78582738abced2e628720225affc778df95"} Nov 03 22:40:11.198561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:11.198524 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:11.262248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:11.262161 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-openshift-ca\") pod \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " Nov 03 22:40:11.262248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:11.262222 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-xks-ca\") pod \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " Nov 03 22:40:11.262502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:11.262286 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-kube-api-access-h7tzf\" (UniqueName: \"kubernetes.io/projected/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-kube-api-access-h7tzf\") pod \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\" (UID: \"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e\") " Nov 03 22:40:11.263050 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:11.263006 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-openshift-ca" pod "b27c49d4-2222-4f4b-9b4f-e1734b0ea57e" (UID: "b27c49d4-2222-4f4b-9b4f-e1734b0ea57e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:11.263409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:11.263392 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:11.264788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:11.264759 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-xks-ca" pod "b27c49d4-2222-4f4b-9b4f-e1734b0ea57e" (UID: "b27c49d4-2222-4f4b-9b4f-e1734b0ea57e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:40:11.265088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:11.265069 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-kube-api-access-h7tzf" pod "b27c49d4-2222-4f4b-9b4f-e1734b0ea57e" (UID: "b27c49d4-2222-4f4b-9b4f-e1734b0ea57e"). InnerVolumeSpecName "kube-api-access-h7tzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:11.364478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:11.364440 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:11.364478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:11.364471 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7tzf\" (UniqueName: \"kubernetes.io/projected/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e-kube-api-access-h7tzf\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:12.000020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:11.999988 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" Nov 03 22:40:12.000208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:12.000020 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk" event={"ID":"b27c49d4-2222-4f4b-9b4f-e1734b0ea57e","Type":"ContainerDied","Data":"3a9245deb36b86d471bf12afd557945e3f7be8e844590a7b965b698fb5adecaa"} Nov 03 22:40:12.000208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:12.000074 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a9245deb36b86d471bf12afd557945e3f7be8e844590a7b965b698fb5adecaa" Nov 03 22:40:12.473613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:12.473581 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:14.372076 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:14.372041 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:16.033545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:16.033463 4298 generic.go:356] "Generic (PLEG): container finished" podID="3da18026-d752-43f1-a018-99c54759fb56" containerID="58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb" exitCode=0 Nov 03 22:40:16.034078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:16.033540 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" event={"ID":"3da18026-d752-43f1-a018-99c54759fb56","Type":"ContainerDied","Data":"58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb"} Nov 03 22:40:16.034148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:16.034107 4298 scope.go:122] "RemoveContainer" containerID="58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb" Nov 03 22:40:16.035919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:16.035892 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:40:16.036000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:16.035958 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:40:16.036000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:16.035989 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:40:17.054922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:17.054889 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" event={"ID":"3da18026-d752-43f1-a018-99c54759fb56","Type":"ContainerStarted","Data":"2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1"} Nov 03 22:40:17.401536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:17.401447 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:17.578157 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:40:17.578121 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:40:18.001975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:18.001931 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:40:18.003562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:18.003536 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:40:18.003734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:18.003592 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 22:40:18.523352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:18.523308 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:40:19.001588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:19.001550 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:40:19.001810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:19.001618 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 22:40:19.037688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:19.033884 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:20.001261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:20.001222 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:40:20.001699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:20.001276 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 22:40:21.001361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:21.001325 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:40:21.001796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:21.001390 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 22:40:21.167420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:21.167387 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:21.352928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:21.352836 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:22.002002 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:22.001968 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:40:22.002502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:22.002019 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 22:40:22.189328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:22.189295 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:23.006610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:23.006576 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:40:23.007943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:23.007922 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:40:24.369977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:24.369944 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:25.472724 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:25.472688 4298 scope.go:122] "RemoveContainer" containerID="97eb309c417676a1230761c070d3c8aece53712a6781f51ea5a5cc1ce492ec53" Nov 03 22:40:27.148943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:27.148897 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:31.554525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:31.554489 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:31.756665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:31.753312 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:32.094998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:32.094962 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:36.243663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:36.238256 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:40.736725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:40.736691 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:41.318775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:41.318739 4298 generic.go:356] "Generic (PLEG): container finished" podID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerID="0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef" exitCode=0 Nov 03 22:40:41.318957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:41.318808 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" event={"ID":"68cf1bc3-4867-4446-866b-249d8cb2c7c3","Type":"ContainerDied","Data":"0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef"} Nov 03 22:40:41.319300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:41.319287 4298 scope.go:122] "RemoveContainer" containerID="0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef" Nov 03 22:40:41.321077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:41.321050 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:40:41.321154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:41.321120 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:40:41.321193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:41.321161 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:40:42.354260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:42.354227 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" event={"ID":"68cf1bc3-4867-4446-866b-249d8cb2c7c3","Type":"ContainerStarted","Data":"d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c"} Nov 03 22:40:42.442685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:42.437553 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:40:42.442685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:42.437700 4298 patch_prober.go:29] interesting pod/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 22:40:42.442685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:42.437747 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 22:40:43.447667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:43.447545 4298 patch_prober.go:29] interesting pod/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 22:40:43.447667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:43.447665 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 22:40:44.443115 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:44.443071 4298 patch_prober.go:29] interesting pod/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 22:40:44.443422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:44.443389 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 22:40:45.273156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:45.273113 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:45.440100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:45.440059 4298 patch_prober.go:29] interesting pod/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 22:40:45.440423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:45.440384 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 22:40:46.449189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:46.449145 4298 patch_prober.go:29] interesting pod/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 22:40:46.449679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:46.449221 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 22:40:46.507600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:46.507573 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:40:47.440666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:47.435036 4298 patch_prober.go:29] interesting pod/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 22:40:47.440666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:47.435100 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 22:40:47.555997 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:40:47.555951 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:40:48.438400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:48.438363 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:40:48.439319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:48.439297 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:40:49.826120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:49.824618 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:52.049105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:52.049070 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq"] Nov 03 22:40:52.099039 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:52.099005 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b90a2563-pk7nq"] Nov 03 22:40:53.137408 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.137369 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8582c1e0-3305-4331-9a89-6aa46a71ae1b" path="/var/lib/kubelet/pods/8582c1e0-3305-4331-9a89-6aa46a71ae1b/volumes" Nov 03 22:40:53.296106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.296076 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s"] Nov 03 22:40:53.296814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.296798 4298 cpu_manager.go:414] "containerMap: removing container" podUID="b27c49d4-2222-4f4b-9b4f-e1734b0ea57e" containerName="revoke-keys" Nov 03 22:40:53.296876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.296815 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27c49d4-2222-4f4b-9b4f-e1734b0ea57e" containerName="revoke-keys" Nov 03 22:40:53.296937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.296928 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27c49d4-2222-4f4b-9b4f-e1734b0ea57e" containerName="revoke-keys" Nov 03 22:40:53.300737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.300718 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:53.357082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.357043 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s"] Nov 03 22:40:53.401758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.401675 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:40:53.401758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.401751 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:40:53.429564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.429522 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca6853-67v6s\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:53.429771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.429573 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwxl\" (UniqueName: \"kubernetes.io/projected/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-kube-api-access-mjwxl\") pod \"maas-api-revoke-keys-e2e-api-a-ca6853-67v6s\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:53.429771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.429606 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca6853-67v6s\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:53.530965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.530924 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwxl\" (UniqueName: \"kubernetes.io/projected/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-kube-api-access-mjwxl\") pod \"maas-api-revoke-keys-e2e-api-a-ca6853-67v6s\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:53.531148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.530988 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca6853-67v6s\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:53.531292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.531271 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca6853-67v6s\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:53.531770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.531752 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca6853-67v6s\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:53.533760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.533738 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca6853-67v6s\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:53.588065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.588025 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwxl\" (UniqueName: \"kubernetes.io/projected/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-kube-api-access-mjwxl\") pod \"maas-api-revoke-keys-e2e-api-a-ca6853-67v6s\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:53.616087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.616059 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:53.988305 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:40:53.988254 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b0b26c_488c_4e62_8f5f_603d0e3d7e98.slice/crio-6bcbffd5928c64ee76c3ddb2482d6b296d9eb19df86fe5005c357edb6dc98cdc WatchSource:0}: Error finding container 6bcbffd5928c64ee76c3ddb2482d6b296d9eb19df86fe5005c357edb6dc98cdc: Status 404 returned error can't find the container with id 6bcbffd5928c64ee76c3ddb2482d6b296d9eb19df86fe5005c357edb6dc98cdc Nov 03 22:40:53.998929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:53.998344 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s"] Nov 03 22:40:54.467008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:54.466891 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:54.476178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:54.476060 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" event={"ID":"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98","Type":"ContainerStarted","Data":"6bcbffd5928c64ee76c3ddb2482d6b296d9eb19df86fe5005c357edb6dc98cdc"} Nov 03 22:40:54.660667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:54.660577 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:54.948542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:54.948512 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:55.485290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:55.485257 4298 generic.go:356] "Generic (PLEG): container finished" podID="b6b0b26c-488c-4e62-8f5f-603d0e3d7e98" containerID="79ccbf97a1a8aca1bef81b6db1e3d8e61e0ae75db8c87e6b076a031d438ff855" exitCode=0 Nov 03 22:40:55.485803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:55.485345 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" event={"ID":"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98","Type":"ContainerDied","Data":"79ccbf97a1a8aca1bef81b6db1e3d8e61e0ae75db8c87e6b076a031d438ff855"} Nov 03 22:40:56.756096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:56.756069 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:56.781400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:56.781364 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-kube-api-access-mjwxl\" (UniqueName: \"kubernetes.io/projected/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-kube-api-access-mjwxl\") pod \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " Nov 03 22:40:56.781577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:56.781460 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-openshift-ca\") pod \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " Nov 03 22:40:56.781635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:56.781617 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-xks-ca\") pod \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\" (UID: \"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98\") " Nov 03 22:40:56.781800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:56.781776 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-openshift-ca" pod "b6b0b26c-488c-4e62-8f5f-603d0e3d7e98" (UID: "b6b0b26c-488c-4e62-8f5f-603d0e3d7e98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:56.782223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:56.782206 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:56.783686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:56.783619 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-xks-ca" pod "b6b0b26c-488c-4e62-8f5f-603d0e3d7e98" (UID: "b6b0b26c-488c-4e62-8f5f-603d0e3d7e98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:40:56.783841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:56.783822 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-kube-api-access-mjwxl" pod "b6b0b26c-488c-4e62-8f5f-603d0e3d7e98" (UID: "b6b0b26c-488c-4e62-8f5f-603d0e3d7e98"). InnerVolumeSpecName "kube-api-access-mjwxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:56.883969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:56.883932 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:56.883969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:56.883970 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjwxl\" (UniqueName: \"kubernetes.io/projected/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98-kube-api-access-mjwxl\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:57.554578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:57.554546 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" event={"ID":"b6b0b26c-488c-4e62-8f5f-603d0e3d7e98","Type":"ContainerDied","Data":"6bcbffd5928c64ee76c3ddb2482d6b296d9eb19df86fe5005c357edb6dc98cdc"} Nov 03 22:40:57.554772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:57.554597 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bcbffd5928c64ee76c3ddb2482d6b296d9eb19df86fe5005c357edb6dc98cdc" Nov 03 22:40:57.554772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:57.554560 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s" Nov 03 22:40:59.569664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:40:59.564381 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:04.421798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:04.421757 4298 patch_prober.go:29] interesting pod/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:41:04.422296 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:04.421833 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:04.470763 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:04.470728 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:04.889359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:04.889326 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:05.072552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:05.072515 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:06.186385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.186353 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp"] Nov 03 22:41:06.187046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.187031 4298 cpu_manager.go:414] "containerMap: removing container" podUID="b6b0b26c-488c-4e62-8f5f-603d0e3d7e98" containerName="revoke-keys" Nov 03 22:41:06.187085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.187048 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b0b26c-488c-4e62-8f5f-603d0e3d7e98" containerName="revoke-keys" Nov 03 22:41:06.187167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.187158 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b0b26c-488c-4e62-8f5f-603d0e3d7e98" containerName="revoke-keys" Nov 03 22:41:06.191119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.191105 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:06.251481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.251430 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp"] Nov 03 22:41:06.266545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.265139 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:06.266545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.265320 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:06.266545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.265443 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlssz\" (UniqueName: \"kubernetes.io/projected/ab1d2964-f777-435a-aeef-c2e2fe843712-kube-api-access-qlssz\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:06.292636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.292602 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:41:06.292863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.292738 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:41:06.369134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.367124 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:06.369134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.367313 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlssz\" (UniqueName: \"kubernetes.io/projected/ab1d2964-f777-435a-aeef-c2e2fe843712-kube-api-access-qlssz\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:06.369134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.367464 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:06.369134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.368814 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:06.382705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.373429 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:06.427338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.427299 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlssz\" (UniqueName: \"kubernetes.io/projected/ab1d2964-f777-435a-aeef-c2e2fe843712-kube-api-access-qlssz\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:06.505839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.505812 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:06.632272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.632233 4298 generic.go:356] "Generic (PLEG): container finished" podID="d8131b38-abea-4933-91b4-e217facd68d8" containerID="4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74" exitCode=0 Nov 03 22:41:06.632481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.632452 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" event={"ID":"d8131b38-abea-4933-91b4-e217facd68d8","Type":"ContainerDied","Data":"4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74"} Nov 03 22:41:06.632962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.632916 4298 scope.go:122] "RemoveContainer" containerID="4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74" Nov 03 22:41:06.635543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.634919 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:06.635543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.635013 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:06.635543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.635058 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:06.737506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.737452 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:06.899356 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:41:06.899313 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1d2964_f777_435a_aeef_c2e2fe843712.slice/crio-a40b66306844ebf3e213f716b08b4127c826757969e690951cb5441c576cabfe WatchSource:0}: Error finding container a40b66306844ebf3e213f716b08b4127c826757969e690951cb5441c576cabfe: Status 404 returned error can't find the container with id a40b66306844ebf3e213f716b08b4127c826757969e690951cb5441c576cabfe Nov 03 22:41:06.905325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.905303 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp"] Nov 03 22:41:06.912289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:06.912266 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:07.653764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.653660 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" event={"ID":"ab1d2964-f777-435a-aeef-c2e2fe843712","Type":"ContainerStarted","Data":"df0d25fc68a35cac59b0afb1ff4ce99609a9a0b038c66dbfab14061d2a0842ed"} Nov 03 22:41:07.653764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.653737 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" event={"ID":"ab1d2964-f777-435a-aeef-c2e2fe843712","Type":"ContainerStarted","Data":"a40b66306844ebf3e213f716b08b4127c826757969e690951cb5441c576cabfe"} Nov 03 22:41:07.655989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.655961 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" event={"ID":"d8131b38-abea-4933-91b4-e217facd68d8","Type":"ContainerStarted","Data":"cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c"} Nov 03 22:41:07.748668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.743854 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz"] Nov 03 22:41:07.751167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.750160 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:07.768995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.768944 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" podStartSLOduration=1.7689289540000002 podStartE2EDuration="1.768928954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:07.766802205 +0000 UTC m=+2024.747535857" watchObservedRunningTime="2026-11-03 22:41:07.768928954 +0000 UTC m=+2024.749662606" Nov 03 22:41:07.797771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.797736 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz"] Nov 03 22:41:07.890832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.890792 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:07.891037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.890932 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:07.891037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.890998 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jncq\" (UniqueName: \"kubernetes.io/projected/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-kube-api-access-7jncq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:07.993367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.993315 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:07.993576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.993430 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:07.993576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.993451 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jncq\" (UniqueName: \"kubernetes.io/projected/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-kube-api-access-7jncq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:07.994091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.994055 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:07.995815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:07.995782 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:08.053100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:08.053065 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jncq\" (UniqueName: \"kubernetes.io/projected/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-kube-api-access-7jncq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:08.068131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:08.068101 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:08.455088 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:41:08.455060 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0bb375_63ee_4abd_a41b_a2d03fd9a793.slice/crio-9f1b71f2e43013eac1d630f9b7821d45f9a8e4765ac2c581af139075d0ab0277 WatchSource:0}: Error finding container 9f1b71f2e43013eac1d630f9b7821d45f9a8e4765ac2c581af139075d0ab0277: Status 404 returned error can't find the container with id 9f1b71f2e43013eac1d630f9b7821d45f9a8e4765ac2c581af139075d0ab0277 Nov 03 22:41:08.456134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:08.456111 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz"] Nov 03 22:41:08.568590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:08.568557 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:41:08.569046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:08.569025 4298 patch_prober.go:29] interesting pod/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 22:41:08.569100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:08.569080 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 22:41:08.665057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:08.665025 4298 generic.go:356] "Generic (PLEG): container finished" podID="ab1d2964-f777-435a-aeef-c2e2fe843712" containerID="df0d25fc68a35cac59b0afb1ff4ce99609a9a0b038c66dbfab14061d2a0842ed" exitCode=0 Nov 03 22:41:08.665436 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:08.665106 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" event={"ID":"ab1d2964-f777-435a-aeef-c2e2fe843712","Type":"ContainerDied","Data":"df0d25fc68a35cac59b0afb1ff4ce99609a9a0b038c66dbfab14061d2a0842ed"} Nov 03 22:41:08.666482 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:08.666457 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" event={"ID":"ff0bb375-63ee-4abd-a41b-a2d03fd9a793","Type":"ContainerStarted","Data":"9f1b71f2e43013eac1d630f9b7821d45f9a8e4765ac2c581af139075d0ab0277"} Nov 03 22:41:09.569184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:09.569153 4298 patch_prober.go:29] interesting pod/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 22:41:09.569368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:09.569206 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 22:41:09.608668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:09.606609 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:09.677138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:09.677103 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" event={"ID":"ff0bb375-63ee-4abd-a41b-a2d03fd9a793","Type":"ContainerStarted","Data":"71a8f9359c28dcbacf779f8fb7f8676c2d83c7a9845e47e9d368436d38c3409a"} Nov 03 22:41:09.863742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:09.845382 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:09.955842 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:09.955815 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:10.048671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.043661 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:10.051480 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.051453 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1d2964-f777-435a-aeef-c2e2fe843712-kube-api-access-qlssz\" (UniqueName: \"kubernetes.io/projected/ab1d2964-f777-435a-aeef-c2e2fe843712-kube-api-access-qlssz\") pod \"ab1d2964-f777-435a-aeef-c2e2fe843712\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " Nov 03 22:41:10.051585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.051547 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-xks-ca\") pod \"ab1d2964-f777-435a-aeef-c2e2fe843712\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " Nov 03 22:41:10.051585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.051578 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-openshift-ca\") pod \"ab1d2964-f777-435a-aeef-c2e2fe843712\" (UID: \"ab1d2964-f777-435a-aeef-c2e2fe843712\") " Nov 03 22:41:10.051994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.051970 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-openshift-ca" pod "ab1d2964-f777-435a-aeef-c2e2fe843712" (UID: "ab1d2964-f777-435a-aeef-c2e2fe843712"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:10.052270 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.052250 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:10.053715 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.053690 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-xks-ca" pod "ab1d2964-f777-435a-aeef-c2e2fe843712" (UID: "ab1d2964-f777-435a-aeef-c2e2fe843712"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:10.053715 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.053702 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1d2964-f777-435a-aeef-c2e2fe843712-kube-api-access-qlssz" pod "ab1d2964-f777-435a-aeef-c2e2fe843712" (UID: "ab1d2964-f777-435a-aeef-c2e2fe843712"). InnerVolumeSpecName "kube-api-access-qlssz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:10.072402 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.072186 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" podStartSLOduration=3.0721667520000002 podStartE2EDuration="3.072166752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:09.809950844 +0000 UTC m=+2026.790684498" watchObservedRunningTime="2026-11-03 22:41:10.072166752 +0000 UTC m=+2027.052900407" Nov 03 22:41:10.153947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.153861 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlssz\" (UniqueName: \"kubernetes.io/projected/ab1d2964-f777-435a-aeef-c2e2fe843712-kube-api-access-qlssz\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:10.153947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.153895 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1d2964-f777-435a-aeef-c2e2fe843712-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:10.569311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.569276 4298 patch_prober.go:29] interesting pod/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 22:41:10.569493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.569330 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 22:41:10.686103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.686067 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" event={"ID":"ab1d2964-f777-435a-aeef-c2e2fe843712","Type":"ContainerDied","Data":"a40b66306844ebf3e213f716b08b4127c826757969e690951cb5441c576cabfe"} Nov 03 22:41:10.686606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.686158 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a40b66306844ebf3e213f716b08b4127c826757969e690951cb5441c576cabfe" Nov 03 22:41:10.686606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.686081 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp" Nov 03 22:41:10.687935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.687904 4298 generic.go:356] "Generic (PLEG): container finished" podID="ff0bb375-63ee-4abd-a41b-a2d03fd9a793" containerID="71a8f9359c28dcbacf779f8fb7f8676c2d83c7a9845e47e9d368436d38c3409a" exitCode=0 Nov 03 22:41:10.688147 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:10.688129 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" event={"ID":"ff0bb375-63ee-4abd-a41b-a2d03fd9a793","Type":"ContainerDied","Data":"71a8f9359c28dcbacf779f8fb7f8676c2d83c7a9845e47e9d368436d38c3409a"} Nov 03 22:41:11.570702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:11.568677 4298 patch_prober.go:29] interesting pod/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 22:41:11.570702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:11.568734 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 22:41:11.964910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:11.964881 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:12.073115 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.073075 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6"] Nov 03 22:41:12.073297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.073271 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" containerName="istio-proxy" containerID="cri-o://8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89" gracePeriod=30 Nov 03 22:41:12.082041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.082016 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-openshift-ca\") pod \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " Nov 03 22:41:12.082205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.082090 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-xks-ca\") pod \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " Nov 03 22:41:12.082375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.082355 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-kube-api-access-7jncq\" (UniqueName: \"kubernetes.io/projected/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-kube-api-access-7jncq\") pod \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\" (UID: \"ff0bb375-63ee-4abd-a41b-a2d03fd9a793\") " Nov 03 22:41:12.082449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.082411 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-openshift-ca" pod "ff0bb375-63ee-4abd-a41b-a2d03fd9a793" (UID: "ff0bb375-63ee-4abd-a41b-a2d03fd9a793"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:12.082937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.082917 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:12.084079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.084055 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-xks-ca" pod "ff0bb375-63ee-4abd-a41b-a2d03fd9a793" (UID: "ff0bb375-63ee-4abd-a41b-a2d03fd9a793"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:12.084262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.084245 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-kube-api-access-7jncq" pod "ff0bb375-63ee-4abd-a41b-a2d03fd9a793" (UID: "ff0bb375-63ee-4abd-a41b-a2d03fd9a793"). InnerVolumeSpecName "kube-api-access-7jncq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:12.190798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.183970 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jncq\" (UniqueName: \"kubernetes.io/projected/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-kube-api-access-7jncq\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:12.190798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.184006 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0bb375-63ee-4abd-a41b-a2d03fd9a793-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:12.196542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.196506 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb"] Nov 03 22:41:12.196816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.196793 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" podUID="948f4945-f3f1-4308-b010-846ab3a4616b" containerName="discovery" containerID="cri-o://afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317" gracePeriod=30 Nov 03 22:41:12.350543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.350504 4298 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dcncb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:8080/ready\": dial tcp 10.133.0.130:8080: connect: connection refused" start-of-body= Nov 03 22:41:12.350624 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.350577 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" podUID="948f4945-f3f1-4308-b010-846ab3a4616b" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.130:8080/ready\": dial tcp 10.133.0.130:8080: connect: connection refused" Nov 03 22:41:12.575678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.574030 4298 patch_prober.go:29] interesting pod/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 22:41:12.575678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.574090 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 22:41:12.594248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.594219 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:41:12.692728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.692698 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-istio-token\" (UniqueName: \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-istio-token\") pod \"948f4945-f3f1-4308-b010-846ab3a4616b\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " Nov 03 22:41:12.692921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.692749 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948f4945-f3f1-4308-b010-846ab3a4616b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/948f4945-f3f1-4308-b010-846ab3a4616b-local-certs\") pod \"948f4945-f3f1-4308-b010-846ab3a4616b\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " Nov 03 22:41:12.692921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.692794 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-ca-configmap\") pod \"948f4945-f3f1-4308-b010-846ab3a4616b\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " Nov 03 22:41:12.692921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.692869 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-dns-cert\") pod \"948f4945-f3f1-4308-b010-846ab3a4616b\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " Nov 03 22:41:12.692921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.692914 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-kubeconfig\") pod \"948f4945-f3f1-4308-b010-846ab3a4616b\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " Nov 03 22:41:12.693127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.693005 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-cacerts\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-cacerts\") pod \"948f4945-f3f1-4308-b010-846ab3a4616b\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " Nov 03 22:41:12.693127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.693035 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-kube-api-access-ptwfb\" (UniqueName: \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-kube-api-access-ptwfb\") pod \"948f4945-f3f1-4308-b010-846ab3a4616b\" (UID: \"948f4945-f3f1-4308-b010-846ab3a4616b\") " Nov 03 22:41:12.693384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.693336 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-ca-configmap" pod "948f4945-f3f1-4308-b010-846ab3a4616b" (UID: "948f4945-f3f1-4308-b010-846ab3a4616b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:12.693903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.693877 4298 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-ca-configmap\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:12.695849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.695816 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-kube-api-access-ptwfb" pod "948f4945-f3f1-4308-b010-846ab3a4616b" (UID: "948f4945-f3f1-4308-b010-846ab3a4616b"). InnerVolumeSpecName "kube-api-access-ptwfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:12.695962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.695858 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-kubeconfig" pod "948f4945-f3f1-4308-b010-846ab3a4616b" (UID: "948f4945-f3f1-4308-b010-846ab3a4616b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:12.695962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.695933 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-dns-cert" pod "948f4945-f3f1-4308-b010-846ab3a4616b" (UID: "948f4945-f3f1-4308-b010-846ab3a4616b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:12.696123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.696103 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948f4945-f3f1-4308-b010-846ab3a4616b-local-certs" pod "948f4945-f3f1-4308-b010-846ab3a4616b" (UID: "948f4945-f3f1-4308-b010-846ab3a4616b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:12.696305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.696282 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-cacerts" pod "948f4945-f3f1-4308-b010-846ab3a4616b" (UID: "948f4945-f3f1-4308-b010-846ab3a4616b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:12.696425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.696353 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-istio-token" pod "948f4945-f3f1-4308-b010-846ab3a4616b" (UID: "948f4945-f3f1-4308-b010-846ab3a4616b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:12.707516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.707492 4298 generic.go:356] "Generic (PLEG): container finished" podID="948f4945-f3f1-4308-b010-846ab3a4616b" containerID="afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317" exitCode=0 Nov 03 22:41:12.707620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.707541 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" Nov 03 22:41:12.707620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.707572 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" event={"ID":"948f4945-f3f1-4308-b010-846ab3a4616b","Type":"ContainerDied","Data":"afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317"} Nov 03 22:41:12.707738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.707624 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb" event={"ID":"948f4945-f3f1-4308-b010-846ab3a4616b","Type":"ContainerDied","Data":"43a5255078c484268528477577d43484f3053e3e906ec1ba07c62b00b3b75778"} Nov 03 22:41:12.707738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.707671 4298 scope.go:122] "RemoveContainer" containerID="afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317" Nov 03 22:41:12.709158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.709052 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" Nov 03 22:41:12.709158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.709100 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz" event={"ID":"ff0bb375-63ee-4abd-a41b-a2d03fd9a793","Type":"ContainerDied","Data":"9f1b71f2e43013eac1d630f9b7821d45f9a8e4765ac2c581af139075d0ab0277"} Nov 03 22:41:12.709158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.709141 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f1b71f2e43013eac1d630f9b7821d45f9a8e4765ac2c581af139075d0ab0277" Nov 03 22:41:12.756968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.756842 4298 scope.go:122] "RemoveContainer" containerID="afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317" Nov 03 22:41:12.757322 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:41:12.757298 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317\": container with ID starting with afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317 not found: ID does not exist" containerID="afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317" Nov 03 22:41:12.757426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.757330 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317"} err="failed to get container status \"afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317\": rpc error: code = NotFound desc = could not find container \"afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317\": container with ID starting with afc53d9cf6ca80c6e625b3c1edc40a34db3648fcc2c01637d38d1ece4627d317 not found: ID does not exist" Nov 03 22:41:12.787720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.787682 4298 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:41:12.787885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.787737 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:41:12.795627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.795600 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:12.795627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.795628 4298 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/948f4945-f3f1-4308-b010-846ab3a4616b-local-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:12.795793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.795653 4298 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-csr-dns-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:12.795793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.795662 4298 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-istio-kubeconfig\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:12.795793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.795672 4298 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/948f4945-f3f1-4308-b010-846ab3a4616b-cacerts\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:12.795793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.795682 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwfb\" (UniqueName: \"kubernetes.io/projected/948f4945-f3f1-4308-b010-846ab3a4616b-kube-api-access-ptwfb\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:12.892204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.892117 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb"] Nov 03 22:41:12.897087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:12.897062 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcncb"] Nov 03 22:41:13.142720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:13.140638 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948f4945-f3f1-4308-b010-846ab3a4616b" path="/var/lib/kubelet/pods/948f4945-f3f1-4308-b010-846ab3a4616b/volumes" Nov 03 22:41:13.574571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:13.574537 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:41:13.576263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:13.575444 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:41:13.577444 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:13.577422 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:41:13.980513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:13.980477 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:14.810886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:14.810545 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:15.571424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:15.571395 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:15.795000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:15.794967 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:15.985682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:15.984275 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:16.111490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:16.111459 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk"] Nov 03 22:41:16.161474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:16.161439 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-37b70b-d87xk"] Nov 03 22:41:16.431543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:16.431462 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:16.876448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:16.876415 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:17.052791 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.052755 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:17.138127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.138037 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b27c49d4-2222-4f4b-9b4f-e1734b0ea57e" path="/var/lib/kubelet/pods/b27c49d4-2222-4f4b-9b4f-e1734b0ea57e/volumes" Nov 03 22:41:17.350463 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.350437 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:41:17.375632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.375602 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-kube-api-access-2622k\" (UniqueName: \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-kube-api-access-2622k\") pod \"cd50b56c-786a-48d9-a89a-431ab176d151\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " Nov 03 22:41:17.375839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.375684 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-istio-token\" (UniqueName: \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-istio-token\") pod \"cd50b56c-786a-48d9-a89a-431ab176d151\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " Nov 03 22:41:17.375839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.375724 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-socket\") pod \"cd50b56c-786a-48d9-a89a-431ab176d151\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " Nov 03 22:41:17.375839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.375754 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-envoy\") pod \"cd50b56c-786a-48d9-a89a-431ab176d151\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " Nov 03 22:41:17.375839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.375800 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd50b56c-786a-48d9-a89a-431ab176d151-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd50b56c-786a-48d9-a89a-431ab176d151-istio-podinfo\") pod \"cd50b56c-786a-48d9-a89a-431ab176d151\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " Nov 03 22:41:17.376066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.375896 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-credential-socket\") pod \"cd50b56c-786a-48d9-a89a-431ab176d151\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " Nov 03 22:41:17.376066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.375974 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-certs\") pod \"cd50b56c-786a-48d9-a89a-431ab176d151\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " Nov 03 22:41:17.376066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.376038 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-socket" pod "cd50b56c-786a-48d9-a89a-431ab176d151" (UID: "cd50b56c-786a-48d9-a89a-431ab176d151"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:17.376233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.376050 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd50b56c-786a-48d9-a89a-431ab176d151-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd50b56c-786a-48d9-a89a-431ab176d151-istiod-ca-cert\") pod \"cd50b56c-786a-48d9-a89a-431ab176d151\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " Nov 03 22:41:17.376322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.376284 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-credential-socket" pod "cd50b56c-786a-48d9-a89a-431ab176d151" (UID: "cd50b56c-786a-48d9-a89a-431ab176d151"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:17.376322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.376308 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-data\") pod \"cd50b56c-786a-48d9-a89a-431ab176d151\" (UID: \"cd50b56c-786a-48d9-a89a-431ab176d151\") " Nov 03 22:41:17.376449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.376376 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd50b56c-786a-48d9-a89a-431ab176d151-istiod-ca-cert" pod "cd50b56c-786a-48d9-a89a-431ab176d151" (UID: "cd50b56c-786a-48d9-a89a-431ab176d151"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:17.376524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.376505 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-data" pod "cd50b56c-786a-48d9-a89a-431ab176d151" (UID: "cd50b56c-786a-48d9-a89a-431ab176d151"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:17.376591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.376524 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-certs" pod "cd50b56c-786a-48d9-a89a-431ab176d151" (UID: "cd50b56c-786a-48d9-a89a-431ab176d151"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:17.377062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.377036 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:17.377175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.377065 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:17.377175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.377079 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:17.377175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.377094 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:17.377175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.377108 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd50b56c-786a-48d9-a89a-431ab176d151-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:17.378195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.378159 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-envoy" pod "cd50b56c-786a-48d9-a89a-431ab176d151" (UID: "cd50b56c-786a-48d9-a89a-431ab176d151"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:17.378389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.378366 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-istio-token" pod "cd50b56c-786a-48d9-a89a-431ab176d151" (UID: "cd50b56c-786a-48d9-a89a-431ab176d151"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:17.378467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.378369 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd50b56c-786a-48d9-a89a-431ab176d151-istio-podinfo" pod "cd50b56c-786a-48d9-a89a-431ab176d151" (UID: "cd50b56c-786a-48d9-a89a-431ab176d151"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:41:17.378467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.378439 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-kube-api-access-2622k" pod "cd50b56c-786a-48d9-a89a-431ab176d151" (UID: "cd50b56c-786a-48d9-a89a-431ab176d151"). InnerVolumeSpecName "kube-api-access-2622k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:17.478349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.478305 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:17.478349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.478338 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd50b56c-786a-48d9-a89a-431ab176d151-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:17.478349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.478353 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd50b56c-786a-48d9-a89a-431ab176d151-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:17.478685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.478362 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2622k\" (UniqueName: \"kubernetes.io/projected/cd50b56c-786a-48d9-a89a-431ab176d151-kube-api-access-2622k\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:17.557746 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:41:17.557700 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:41:17.748998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.748921 4298 generic.go:356] "Generic (PLEG): container finished" podID="cd50b56c-786a-48d9-a89a-431ab176d151" containerID="8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89" exitCode=0 Nov 03 22:41:17.748998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.748971 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" Nov 03 22:41:17.749234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.749006 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" event={"ID":"cd50b56c-786a-48d9-a89a-431ab176d151","Type":"ContainerDied","Data":"8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89"} Nov 03 22:41:17.749234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.749065 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6" event={"ID":"cd50b56c-786a-48d9-a89a-431ab176d151","Type":"ContainerDied","Data":"fe56140fa8ade8d2985105fa28badd248b3df2f53a1edabb3998bc05faed9a7c"} Nov 03 22:41:17.749234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.749094 4298 scope.go:122] "RemoveContainer" containerID="8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89" Nov 03 22:41:17.762560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.762540 4298 scope.go:122] "RemoveContainer" containerID="8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89" Nov 03 22:41:17.762885 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:41:17.762867 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89\": container with ID starting with 8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89 not found: ID does not exist" containerID="8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89" Nov 03 22:41:17.762945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.762895 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89"} err="failed to get container status \"8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89\": rpc error: code = NotFound desc = could not find container \"8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89\": container with ID starting with 8b495bd3b766a0c482223452ef47895045d7cc9b43736b0db5a9b22fc51afe89 not found: ID does not exist" Nov 03 22:41:17.916886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.916851 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6"] Nov 03 22:41:17.921726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:17.921699 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-rc4x6"] Nov 03 22:41:19.136326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:19.136294 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" path="/var/lib/kubelet/pods/cd50b56c-786a-48d9-a89a-431ab176d151/volumes" Nov 03 22:41:19.379903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:19.379871 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:20.734685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:20.734620 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:21.054177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:21.054085 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48"] Nov 03 22:41:21.098777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:21.098748 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee9c090c-7tv48"] Nov 03 22:41:21.137319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:21.137284 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed014b7f-5354-4201-b2b5-df5bd0e81c05" path="/var/lib/kubelet/pods/ed014b7f-5354-4201-b2b5-df5bd0e81c05/volumes" Nov 03 22:41:21.455528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:21.455496 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db"] Nov 03 22:41:21.557292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:21.557261 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9"] Nov 03 22:41:21.854898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:21.854820 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:22.118951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:22.118859 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:22.455843 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:22.455812 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:22.645636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:22.645601 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:22.862681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:22.862181 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:23.713795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.713759 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv"] Nov 03 22:41:23.714408 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714395 4298 cpu_manager.go:414] "containerMap: removing container" podUID="948f4945-f3f1-4308-b010-846ab3a4616b" containerName="discovery" Nov 03 22:41:23.714453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714410 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948f4945-f3f1-4308-b010-846ab3a4616b" containerName="discovery" Nov 03 22:41:23.714453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714448 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ab1d2964-f777-435a-aeef-c2e2fe843712" containerName="revoke-keys" Nov 03 22:41:23.714519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714453 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1d2964-f777-435a-aeef-c2e2fe843712" containerName="revoke-keys" Nov 03 22:41:23.714519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714466 4298 cpu_manager.go:414] "containerMap: removing container" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" containerName="istio-proxy" Nov 03 22:41:23.714519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714471 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" containerName="istio-proxy" Nov 03 22:41:23.714519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714478 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ff0bb375-63ee-4abd-a41b-a2d03fd9a793" containerName="revoke-keys" Nov 03 22:41:23.714519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714482 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0bb375-63ee-4abd-a41b-a2d03fd9a793" containerName="revoke-keys" Nov 03 22:41:23.714701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714591 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd50b56c-786a-48d9-a89a-431ab176d151" containerName="istio-proxy" Nov 03 22:41:23.714701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714601 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0bb375-63ee-4abd-a41b-a2d03fd9a793" containerName="revoke-keys" Nov 03 22:41:23.714701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714617 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="948f4945-f3f1-4308-b010-846ab3a4616b" containerName="discovery" Nov 03 22:41:23.714701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.714630 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1d2964-f777-435a-aeef-c2e2fe843712" containerName="revoke-keys" Nov 03 22:41:23.719635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.719616 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:23.775236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.775207 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv"] Nov 03 22:41:23.789663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.789612 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:23.789859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.789691 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzs52\" (UniqueName: \"kubernetes.io/projected/ab38f8d8-386d-4b26-958c-d51d4382180b-kube-api-access-fzs52\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:23.789928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.789881 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:23.821966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.821934 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:41:23.821966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.821947 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:41:23.890973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.890941 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:23.891467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.890982 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzs52\" (UniqueName: \"kubernetes.io/projected/ab38f8d8-386d-4b26-958c-d51d4382180b-kube-api-access-fzs52\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:23.891467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.891096 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:23.891874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.891849 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:23.893569 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.893550 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:23.948125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:23.948090 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzs52\" (UniqueName: \"kubernetes.io/projected/ab38f8d8-386d-4b26-958c-d51d4382180b-kube-api-access-fzs52\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:24.033785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:24.033704 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:24.404981 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:41:24.404951 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab38f8d8_386d_4b26_958c_d51d4382180b.slice/crio-da896f6ca02eaba8dbec4895ebab0b1f382355c86441f57711395b918f79d066 WatchSource:0}: Error finding container da896f6ca02eaba8dbec4895ebab0b1f382355c86441f57711395b918f79d066: Status 404 returned error can't find the container with id da896f6ca02eaba8dbec4895ebab0b1f382355c86441f57711395b918f79d066 Nov 03 22:41:24.410265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:24.410109 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv"] Nov 03 22:41:24.729680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:24.729633 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:24.804852 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:24.804812 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" event={"ID":"ab38f8d8-386d-4b26-958c-d51d4382180b","Type":"ContainerStarted","Data":"7583e1898a340c186f7b4d436d49aec3832786566bdf7c83a58471eab3cfd10e"} Nov 03 22:41:24.805036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:24.804914 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" event={"ID":"ab38f8d8-386d-4b26-958c-d51d4382180b","Type":"ContainerStarted","Data":"da896f6ca02eaba8dbec4895ebab0b1f382355c86441f57711395b918f79d066"} Nov 03 22:41:24.921463 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:24.921407 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" podStartSLOduration=1.921392849 podStartE2EDuration="1.921392849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:24.918356918 +0000 UTC m=+2041.899090573" watchObservedRunningTime="2026-11-03 22:41:24.921392849 +0000 UTC m=+2041.902126502" Nov 03 22:41:25.200133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:25.200096 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:25.511808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:25.511696 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z"] Nov 03 22:41:25.526423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:25.523218 4298 scope.go:122] "RemoveContainer" containerID="939e3bde482500ca1ca6328b8d207179d2cd54c1964e59126c8f2e3550b07642" Nov 03 22:41:25.562533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:25.562506 4298 scope.go:122] "RemoveContainer" containerID="cd333ba3b66bb6ab0cf92b72df8bed700be3a3895bff4984485d395b23d5afa1" Nov 03 22:41:25.609210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:25.609180 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x"] Nov 03 22:41:25.813532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:25.813438 4298 generic.go:356] "Generic (PLEG): container finished" podID="ab38f8d8-386d-4b26-958c-d51d4382180b" containerID="7583e1898a340c186f7b4d436d49aec3832786566bdf7c83a58471eab3cfd10e" exitCode=0 Nov 03 22:41:25.813532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:25.813521 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" event={"ID":"ab38f8d8-386d-4b26-958c-d51d4382180b","Type":"ContainerDied","Data":"7583e1898a340c186f7b4d436d49aec3832786566bdf7c83a58471eab3cfd10e"} Nov 03 22:41:25.814674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:25.814626 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:26.108779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.090497 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:26.456340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.456302 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" podUID="83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de" containerName="payload-processing" containerID="cri-o://78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2" gracePeriod=25 Nov 03 22:41:26.557772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.557721 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" podUID="2359d456-83bd-4d71-8b79-d13bde8ff0da" containerName="payload-pre-processing" containerID="cri-o://d7931fe1cb02e99e1521b45d77ae091ab7dad3d4e639e406aa760d962c1d386a" gracePeriod=25 Nov 03 22:41:26.744318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.744285 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:41:26.823450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.823377 4298 generic.go:356] "Generic (PLEG): container finished" podID="83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de" containerID="78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2" exitCode=0 Nov 03 22:41:26.823450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.823434 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" Nov 03 22:41:26.823450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.823440 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" event={"ID":"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de","Type":"ContainerDied","Data":"78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2"} Nov 03 22:41:26.823735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.823536 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db" event={"ID":"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de","Type":"ContainerDied","Data":"097689c795f978a2d7b1a8e56620010f12ec465cc8620696399a2b0bc26764dd"} Nov 03 22:41:26.823735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.823596 4298 scope.go:122] "RemoveContainer" containerID="78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2" Nov 03 22:41:26.825958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.825672 4298 generic.go:356] "Generic (PLEG): container finished" podID="2359d456-83bd-4d71-8b79-d13bde8ff0da" containerID="d7931fe1cb02e99e1521b45d77ae091ab7dad3d4e639e406aa760d962c1d386a" exitCode=0 Nov 03 22:41:26.825958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.825720 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" event={"ID":"2359d456-83bd-4d71-8b79-d13bde8ff0da","Type":"ContainerDied","Data":"d7931fe1cb02e99e1521b45d77ae091ab7dad3d4e639e406aa760d962c1d386a"} Nov 03 22:41:26.825958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.825807 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" event={"ID":"2359d456-83bd-4d71-8b79-d13bde8ff0da","Type":"ContainerDied","Data":"6f67af5122d17c54ecce7f8ea47aa519f237cdc4a6609b6b744ac6b04e62f150"} Nov 03 22:41:26.825958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.825846 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f67af5122d17c54ecce7f8ea47aa519f237cdc4a6609b6b744ac6b04e62f150" Nov 03 22:41:26.836436 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.836412 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:41:26.837797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.837772 4298 scope.go:122] "RemoveContainer" containerID="78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2" Nov 03 22:41:26.838101 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:41:26.838079 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2\": container with ID starting with 78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2 not found: ID does not exist" containerID="78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2" Nov 03 22:41:26.838181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.838112 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2"} err="failed to get container status \"78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2\": rpc error: code = NotFound desc = could not find container \"78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2\": container with ID starting with 78096e77e6a948c64581b691f16917236aa1cf919786199f042c944441ade5b2 not found: ID does not exist" Nov 03 22:41:26.857310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.857274 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-plugins-config-volume\") pod \"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de\" (UID: \"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de\") " Nov 03 22:41:26.857492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.857439 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-kube-api-access-wv2mq\" (UniqueName: \"kubernetes.io/projected/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-kube-api-access-wv2mq\") pod \"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de\" (UID: \"83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de\") " Nov 03 22:41:26.857846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.857811 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-plugins-config-volume" pod "83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de" (UID: "83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:26.858285 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.858259 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:26.859872 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.859846 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-kube-api-access-wv2mq" pod "83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de" (UID: "83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de"). InnerVolumeSpecName "kube-api-access-wv2mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:26.959661 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.959608 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2359d456-83bd-4d71-8b79-d13bde8ff0da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2359d456-83bd-4d71-8b79-d13bde8ff0da-plugins-config-volume\") pod \"2359d456-83bd-4d71-8b79-d13bde8ff0da\" (UID: \"2359d456-83bd-4d71-8b79-d13bde8ff0da\") " Nov 03 22:41:26.959854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.959776 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2359d456-83bd-4d71-8b79-d13bde8ff0da-kube-api-access-v7xd2\" (UniqueName: \"kubernetes.io/projected/2359d456-83bd-4d71-8b79-d13bde8ff0da-kube-api-access-v7xd2\") pod \"2359d456-83bd-4d71-8b79-d13bde8ff0da\" (UID: \"2359d456-83bd-4d71-8b79-d13bde8ff0da\") " Nov 03 22:41:26.960042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.960014 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2359d456-83bd-4d71-8b79-d13bde8ff0da-plugins-config-volume" pod "2359d456-83bd-4d71-8b79-d13bde8ff0da" (UID: "2359d456-83bd-4d71-8b79-d13bde8ff0da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:26.960421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.960396 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2359d456-83bd-4d71-8b79-d13bde8ff0da-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:26.960421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.960422 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv2mq\" (UniqueName: \"kubernetes.io/projected/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de-kube-api-access-wv2mq\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:26.962153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.962132 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2359d456-83bd-4d71-8b79-d13bde8ff0da-kube-api-access-v7xd2" pod "2359d456-83bd-4d71-8b79-d13bde8ff0da" (UID: "2359d456-83bd-4d71-8b79-d13bde8ff0da"). InnerVolumeSpecName "kube-api-access-v7xd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:26.995376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:26.995347 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:27.061044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.061004 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-xks-ca\") pod \"ab38f8d8-386d-4b26-958c-d51d4382180b\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " Nov 03 22:41:27.061241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.061094 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab38f8d8-386d-4b26-958c-d51d4382180b-kube-api-access-fzs52\" (UniqueName: \"kubernetes.io/projected/ab38f8d8-386d-4b26-958c-d51d4382180b-kube-api-access-fzs52\") pod \"ab38f8d8-386d-4b26-958c-d51d4382180b\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " Nov 03 22:41:27.061320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.061238 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-openshift-ca\") pod \"ab38f8d8-386d-4b26-958c-d51d4382180b\" (UID: \"ab38f8d8-386d-4b26-958c-d51d4382180b\") " Nov 03 22:41:27.061732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.061697 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-openshift-ca" pod "ab38f8d8-386d-4b26-958c-d51d4382180b" (UID: "ab38f8d8-386d-4b26-958c-d51d4382180b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:27.061893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.061869 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:27.061974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.061895 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7xd2\" (UniqueName: \"kubernetes.io/projected/2359d456-83bd-4d71-8b79-d13bde8ff0da-kube-api-access-v7xd2\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:27.063345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.063315 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-xks-ca" pod "ab38f8d8-386d-4b26-958c-d51d4382180b" (UID: "ab38f8d8-386d-4b26-958c-d51d4382180b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:27.063785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.063761 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab38f8d8-386d-4b26-958c-d51d4382180b-kube-api-access-fzs52" pod "ab38f8d8-386d-4b26-958c-d51d4382180b" (UID: "ab38f8d8-386d-4b26-958c-d51d4382180b"). InnerVolumeSpecName "kube-api-access-fzs52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:27.085270 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.085238 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:27.163717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.163682 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab38f8d8-386d-4b26-958c-d51d4382180b-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:27.164065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.163723 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzs52\" (UniqueName: \"kubernetes.io/projected/ab38f8d8-386d-4b26-958c-d51d4382180b-kube-api-access-fzs52\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:27.258420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.258332 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db"] Nov 03 22:41:27.263006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.262984 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f61dc0-77896f4698-hc8db"] Nov 03 22:41:27.839087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.839047 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" event={"ID":"ab38f8d8-386d-4b26-958c-d51d4382180b","Type":"ContainerDied","Data":"da896f6ca02eaba8dbec4895ebab0b1f382355c86441f57711395b918f79d066"} Nov 03 22:41:27.839315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.839150 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da896f6ca02eaba8dbec4895ebab0b1f382355c86441f57711395b918f79d066" Nov 03 22:41:27.839315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.839244 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv" Nov 03 22:41:27.840745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:27.840715 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9" Nov 03 22:41:28.024304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:28.024270 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9"] Nov 03 22:41:28.033935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:28.033907 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f61dc0-85c9849c95-k8qd9"] Nov 03 22:41:29.136312 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:29.136281 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2359d456-83bd-4d71-8b79-d13bde8ff0da" path="/var/lib/kubelet/pods/2359d456-83bd-4d71-8b79-d13bde8ff0da/volumes" Nov 03 22:41:29.136751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:29.136736 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de" path="/var/lib/kubelet/pods/83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de/volumes" Nov 03 22:41:30.512655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.512586 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" podUID="3541be3e-2e43-4741-a8ac-a3cd3d5df25d" containerName="payload-processing" containerID="cri-o://e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72" gracePeriod=25 Nov 03 22:41:30.609544 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.609492 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" podUID="2cd9ab68-16fa-4867-9a8d-11b880e416ca" containerName="payload-pre-processing" containerID="cri-o://88e7f514ee9b641f4ae87153947ba933ddd8320aac455fcade115a7df434e3b6" gracePeriod=25 Nov 03 22:41:30.809655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.809612 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:41:30.862271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.862227 4298 generic.go:356] "Generic (PLEG): container finished" podID="3541be3e-2e43-4741-a8ac-a3cd3d5df25d" containerID="e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72" exitCode=0 Nov 03 22:41:30.862491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.862277 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" Nov 03 22:41:30.862491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.862329 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" event={"ID":"3541be3e-2e43-4741-a8ac-a3cd3d5df25d","Type":"ContainerDied","Data":"e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72"} Nov 03 22:41:30.862491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.862386 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z" event={"ID":"3541be3e-2e43-4741-a8ac-a3cd3d5df25d","Type":"ContainerDied","Data":"cf6b59fdf26260a4030612fee9bbb71d367370ebb9693ae1363f5d2ead1d4607"} Nov 03 22:41:30.862491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.862413 4298 scope.go:122] "RemoveContainer" containerID="e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72" Nov 03 22:41:30.864568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.864545 4298 generic.go:356] "Generic (PLEG): container finished" podID="2cd9ab68-16fa-4867-9a8d-11b880e416ca" containerID="88e7f514ee9b641f4ae87153947ba933ddd8320aac455fcade115a7df434e3b6" exitCode=0 Nov 03 22:41:30.864732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.864602 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" event={"ID":"2cd9ab68-16fa-4867-9a8d-11b880e416ca","Type":"ContainerDied","Data":"88e7f514ee9b641f4ae87153947ba933ddd8320aac455fcade115a7df434e3b6"} Nov 03 22:41:30.873273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.873252 4298 scope.go:122] "RemoveContainer" containerID="e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72" Nov 03 22:41:30.873607 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:41:30.873589 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72\": container with ID starting with e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72 not found: ID does not exist" containerID="e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72" Nov 03 22:41:30.873719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.873619 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72"} err="failed to get container status \"e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72\": rpc error: code = NotFound desc = could not find container \"e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72\": container with ID starting with e7b80c187005996d7bcf7c71cc8708efe75e9c29e46dd9ef5aa2b14c40549d72 not found: ID does not exist" Nov 03 22:41:30.909128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.909105 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:41:30.918401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.918371 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-kube-api-access-zxszs\" (UniqueName: \"kubernetes.io/projected/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-kube-api-access-zxszs\") pod \"3541be3e-2e43-4741-a8ac-a3cd3d5df25d\" (UID: \"3541be3e-2e43-4741-a8ac-a3cd3d5df25d\") " Nov 03 22:41:30.918609 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.918589 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-plugins-config-volume\") pod \"3541be3e-2e43-4741-a8ac-a3cd3d5df25d\" (UID: \"3541be3e-2e43-4741-a8ac-a3cd3d5df25d\") " Nov 03 22:41:30.918955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.918925 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-plugins-config-volume" pod "3541be3e-2e43-4741-a8ac-a3cd3d5df25d" (UID: "3541be3e-2e43-4741-a8ac-a3cd3d5df25d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:30.919199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.919178 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:30.920582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:30.920562 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-kube-api-access-zxszs" pod "3541be3e-2e43-4741-a8ac-a3cd3d5df25d" (UID: "3541be3e-2e43-4741-a8ac-a3cd3d5df25d"). InnerVolumeSpecName "kube-api-access-zxszs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:31.020674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.020538 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd9ab68-16fa-4867-9a8d-11b880e416ca-kube-api-access-km6v8\" (UniqueName: \"kubernetes.io/projected/2cd9ab68-16fa-4867-9a8d-11b880e416ca-kube-api-access-km6v8\") pod \"2cd9ab68-16fa-4867-9a8d-11b880e416ca\" (UID: \"2cd9ab68-16fa-4867-9a8d-11b880e416ca\") " Nov 03 22:41:31.020674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.020594 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd9ab68-16fa-4867-9a8d-11b880e416ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd9ab68-16fa-4867-9a8d-11b880e416ca-plugins-config-volume\") pod \"2cd9ab68-16fa-4867-9a8d-11b880e416ca\" (UID: \"2cd9ab68-16fa-4867-9a8d-11b880e416ca\") " Nov 03 22:41:31.021088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.021049 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd9ab68-16fa-4867-9a8d-11b880e416ca-plugins-config-volume" pod "2cd9ab68-16fa-4867-9a8d-11b880e416ca" (UID: "2cd9ab68-16fa-4867-9a8d-11b880e416ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:31.021279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.021255 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd9ab68-16fa-4867-9a8d-11b880e416ca-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:31.021279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.021277 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxszs\" (UniqueName: \"kubernetes.io/projected/3541be3e-2e43-4741-a8ac-a3cd3d5df25d-kube-api-access-zxszs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:31.022586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.022559 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd9ab68-16fa-4867-9a8d-11b880e416ca-kube-api-access-km6v8" pod "2cd9ab68-16fa-4867-9a8d-11b880e416ca" (UID: "2cd9ab68-16fa-4867-9a8d-11b880e416ca"). InnerVolumeSpecName "kube-api-access-km6v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:31.122788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.122754 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km6v8\" (UniqueName: \"kubernetes.io/projected/2cd9ab68-16fa-4867-9a8d-11b880e416ca-kube-api-access-km6v8\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:31.279734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.279624 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z"] Nov 03 22:41:31.284934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.284904 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-094053-9456b56d6-xtq8z"] Nov 03 22:41:31.872972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.872875 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" Nov 03 22:41:31.873435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.872890 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x" event={"ID":"2cd9ab68-16fa-4867-9a8d-11b880e416ca","Type":"ContainerDied","Data":"79c3970ab518fc74ec585e5c68156e67830a5ea889ce88d2dad4dec42cc8efdf"} Nov 03 22:41:31.873435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:31.873065 4298 scope.go:122] "RemoveContainer" containerID="88e7f514ee9b641f4ae87153947ba933ddd8320aac455fcade115a7df434e3b6" Nov 03 22:41:32.032336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:32.032304 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:32.049448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:32.049403 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x"] Nov 03 22:41:32.054579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:32.054552 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-094053-6d77d5d6d7-g2w5x"] Nov 03 22:41:33.136009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:33.135977 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd9ab68-16fa-4867-9a8d-11b880e416ca" path="/var/lib/kubelet/pods/2cd9ab68-16fa-4867-9a8d-11b880e416ca/volumes" Nov 03 22:41:33.136421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:33.136390 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3541be3e-2e43-4741-a8ac-a3cd3d5df25d" path="/var/lib/kubelet/pods/3541be3e-2e43-4741-a8ac-a3cd3d5df25d/volumes" Nov 03 22:41:33.276799 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:33.276771 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:33.456202 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:33.456162 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:34.186602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:34.186559 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n"] Nov 03 22:41:34.298593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:34.298555 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx"] Nov 03 22:41:34.587139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:34.587109 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:34.791571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:34.791539 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:34.972760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:34.972732 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:35.099087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:35.099052 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz"] Nov 03 22:41:35.103552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:35.103531 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-094053-256qz"] Nov 03 22:41:35.137624 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:35.137597 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff0bb375-63ee-4abd-a41b-a2d03fd9a793" path="/var/lib/kubelet/pods/ff0bb375-63ee-4abd-a41b-a2d03fd9a793/volumes" Nov 03 22:41:35.536654 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:35.536606 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:36.963428 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:36.963391 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:39.187283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.187242 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" podUID="178cb1ca-90dd-43b7-ae38-b2bbe401bd9c" containerName="payload-processing" containerID="cri-o://c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc" gracePeriod=25 Nov 03 22:41:39.299531 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.299476 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" podUID="1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1" containerName="payload-pre-processing" containerID="cri-o://f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31" gracePeriod=25 Nov 03 22:41:39.468731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.468700 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:41:39.598540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.598512 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:41:39.639829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.639798 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-kube-api-access-682mg\" (UniqueName: \"kubernetes.io/projected/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-kube-api-access-682mg\") pod \"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c\" (UID: \"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c\") " Nov 03 22:41:39.640055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.640035 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-plugins-config-volume\") pod \"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c\" (UID: \"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c\") " Nov 03 22:41:39.640424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.640388 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-plugins-config-volume" pod "178cb1ca-90dd-43b7-ae38-b2bbe401bd9c" (UID: "178cb1ca-90dd-43b7-ae38-b2bbe401bd9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:39.640683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.640665 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:39.641881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.641852 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-kube-api-access-682mg" pod "178cb1ca-90dd-43b7-ae38-b2bbe401bd9c" (UID: "178cb1ca-90dd-43b7-ae38-b2bbe401bd9c"). InnerVolumeSpecName "kube-api-access-682mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:39.741822 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.741732 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-plugins-config-volume\") pod \"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1\" (UID: \"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1\") " Nov 03 22:41:39.741978 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.741915 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-kube-api-access-qdh45\" (UniqueName: \"kubernetes.io/projected/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-kube-api-access-qdh45\") pod \"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1\" (UID: \"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1\") " Nov 03 22:41:39.742166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.742138 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-plugins-config-volume" pod "1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1" (UID: "1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:39.742505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.742485 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-682mg\" (UniqueName: \"kubernetes.io/projected/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c-kube-api-access-682mg\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:39.742505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.742504 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:39.744038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.744016 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-kube-api-access-qdh45" pod "1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1" (UID: "1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1"). InnerVolumeSpecName "kube-api-access-qdh45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:39.843775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.843739 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdh45\" (UniqueName: \"kubernetes.io/projected/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1-kube-api-access-qdh45\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:39.938720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.938680 4298 generic.go:356] "Generic (PLEG): container finished" podID="1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1" containerID="f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31" exitCode=0 Nov 03 22:41:39.938927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.938726 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" Nov 03 22:41:39.938927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.938772 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" event={"ID":"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1","Type":"ContainerDied","Data":"f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31"} Nov 03 22:41:39.938927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.938824 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx" event={"ID":"1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1","Type":"ContainerDied","Data":"7e7b5b300a320ba92e0f3833d711989690b784a30d34b7f727fd78b6312285df"} Nov 03 22:41:39.938927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.938849 4298 scope.go:122] "RemoveContainer" containerID="f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31" Nov 03 22:41:39.940393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.940364 4298 generic.go:356] "Generic (PLEG): container finished" podID="178cb1ca-90dd-43b7-ae38-b2bbe401bd9c" containerID="c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc" exitCode=0 Nov 03 22:41:39.940549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.940412 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" Nov 03 22:41:39.940549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.940458 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" event={"ID":"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c","Type":"ContainerDied","Data":"c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc"} Nov 03 22:41:39.940549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.940515 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n" event={"ID":"178cb1ca-90dd-43b7-ae38-b2bbe401bd9c","Type":"ContainerDied","Data":"d06e647858d9258197b4555def8557c6882486317805234fe930d249108985c8"} Nov 03 22:41:39.958455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.952706 4298 scope.go:122] "RemoveContainer" containerID="f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31" Nov 03 22:41:39.959841 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:41:39.958944 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31\": container with ID starting with f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31 not found: ID does not exist" containerID="f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31" Nov 03 22:41:39.959841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.958994 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31"} err="failed to get container status \"f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31\": rpc error: code = NotFound desc = could not find container \"f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31\": container with ID starting with f37addc2426bc0ef96d952d806689e899380397ee3466369c6768220c44c6c31 not found: ID does not exist" Nov 03 22:41:39.959841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.959022 4298 scope.go:122] "RemoveContainer" containerID="c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc" Nov 03 22:41:39.971558 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.971531 4298 scope.go:122] "RemoveContainer" containerID="c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc" Nov 03 22:41:39.971917 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:41:39.971897 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc\": container with ID starting with c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc not found: ID does not exist" containerID="c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc" Nov 03 22:41:39.971975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:39.971928 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc"} err="failed to get container status \"c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc\": rpc error: code = NotFound desc = could not find container \"c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc\": container with ID starting with c37cb9aef4755d73f09b90dcc5188c49b17f0c4138d62650967fbdefd072a9fc not found: ID does not exist" Nov 03 22:41:40.107007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:40.106921 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx"] Nov 03 22:41:40.156953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:40.156918 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a92c9e-6b4bbdb68c-rhksx"] Nov 03 22:41:40.221508 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:40.221473 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n"] Nov 03 22:41:40.228080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:40.228051 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a92c9e-58b8d47fb4-lhw7n"] Nov 03 22:41:41.136165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:41.136134 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178cb1ca-90dd-43b7-ae38-b2bbe401bd9c" path="/var/lib/kubelet/pods/178cb1ca-90dd-43b7-ae38-b2bbe401bd9c/volumes" Nov 03 22:41:41.136544 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:41.136529 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1" path="/var/lib/kubelet/pods/1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1/volumes" Nov 03 22:41:41.404230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:41.404146 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:41.585291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:41.585257 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:41.691517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:41.691435 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h"] Nov 03 22:41:41.691839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:41.691808 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" containerID="cri-o://d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c" gracePeriod=30 Nov 03 22:41:41.767655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:41.767612 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:41.951758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:41.951728 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:43.876785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:43.876754 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:44.833898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.833864 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h"] Nov 03 22:41:44.834491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834476 4298 cpu_manager.go:414] "containerMap: removing container" podUID="2cd9ab68-16fa-4867-9a8d-11b880e416ca" containerName="payload-pre-processing" Nov 03 22:41:44.834533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834493 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd9ab68-16fa-4867-9a8d-11b880e416ca" containerName="payload-pre-processing" Nov 03 22:41:44.834533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834510 4298 cpu_manager.go:414] "containerMap: removing container" podUID="83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de" containerName="payload-processing" Nov 03 22:41:44.834533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834516 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de" containerName="payload-processing" Nov 03 22:41:44.834533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834523 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3541be3e-2e43-4741-a8ac-a3cd3d5df25d" containerName="payload-processing" Nov 03 22:41:44.834533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834528 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3541be3e-2e43-4741-a8ac-a3cd3d5df25d" containerName="payload-processing" Nov 03 22:41:44.834734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834539 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ab38f8d8-386d-4b26-958c-d51d4382180b" containerName="revoke-keys" Nov 03 22:41:44.834734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834544 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab38f8d8-386d-4b26-958c-d51d4382180b" containerName="revoke-keys" Nov 03 22:41:44.834734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834565 4298 cpu_manager.go:414] "containerMap: removing container" podUID="178cb1ca-90dd-43b7-ae38-b2bbe401bd9c" containerName="payload-processing" Nov 03 22:41:44.834734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834570 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178cb1ca-90dd-43b7-ae38-b2bbe401bd9c" containerName="payload-processing" Nov 03 22:41:44.834734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834582 4298 cpu_manager.go:414] "containerMap: removing container" podUID="2359d456-83bd-4d71-8b79-d13bde8ff0da" containerName="payload-pre-processing" Nov 03 22:41:44.834734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834587 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2359d456-83bd-4d71-8b79-d13bde8ff0da" containerName="payload-pre-processing" Nov 03 22:41:44.834734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834598 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1" containerName="payload-pre-processing" Nov 03 22:41:44.834734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834603 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1" containerName="payload-pre-processing" Nov 03 22:41:44.834734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834727 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="178cb1ca-90dd-43b7-ae38-b2bbe401bd9c" containerName="payload-processing" Nov 03 22:41:44.834986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834739 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="2359d456-83bd-4d71-8b79-d13bde8ff0da" containerName="payload-pre-processing" Nov 03 22:41:44.834986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834746 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c647dcc-fd5b-42c5-a7a0-7eabd8b751c1" containerName="payload-pre-processing" Nov 03 22:41:44.834986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834755 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd9ab68-16fa-4867-9a8d-11b880e416ca" containerName="payload-pre-processing" Nov 03 22:41:44.834986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834763 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="83dc4bb3-9757-4aee-9cbf-f6e2eb3d03de" containerName="payload-processing" Nov 03 22:41:44.834986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834768 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3541be3e-2e43-4741-a8ac-a3cd3d5df25d" containerName="payload-processing" Nov 03 22:41:44.834986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.834773 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab38f8d8-386d-4b26-958c-d51d4382180b" containerName="revoke-keys" Nov 03 22:41:44.840090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.840072 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:44.897668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.893807 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f881e3-openshift-default-dockercfg-dfjvx\"" Nov 03 22:41:44.922870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.922834 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-podinfo\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:44.923027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.922894 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-certs\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:44.923027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.922974 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85d19cf-d09f-427c-a04c-f97d95723d99-istiod-ca-cert\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:44.923143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.923086 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-token\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:44.923217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.923199 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-credential-socket\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:44.923354 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.923335 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv26n\" (UniqueName: \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-kube-api-access-xv26n\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:44.923403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.923388 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-data\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:44.923444 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.923431 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-envoy\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:44.923555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.923528 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-socket\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:44.950781 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.950743 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h"] Nov 03 22:41:44.984128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:44.984063 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.025115 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-podinfo\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.025190 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-certs\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.025215 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85d19cf-d09f-427c-a04c-f97d95723d99-istiod-ca-cert\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.025237 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-token\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.025283 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-credential-socket\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.025368 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv26n\" (UniqueName: \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-kube-api-access-xv26n\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.025396 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-data\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.025424 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-envoy\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.025460 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-socket\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.025946 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-socket\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.025991 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-credential-socket\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.026253 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-data\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.026581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.026525 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-certs\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.027412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.027193 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85d19cf-d09f-427c-a04c-f97d95723d99-istiod-ca-cert\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.029594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.029565 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-envoy\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.029854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.029831 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-podinfo\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.086110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.086022 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-token\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.086289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.086189 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv26n\" (UniqueName: \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-kube-api-access-xv26n\") pod \"e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.160679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.160616 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:45.379472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.379335 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:45.587009 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:41:45.586919 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85d19cf_d09f_427c_a04c_f97d95723d99.slice/crio-bb8725ad925552762b8c207ef54ce8c352f73622fb8e91b557bbce01f1e017e8 WatchSource:0}: Error finding container bb8725ad925552762b8c207ef54ce8c352f73622fb8e91b557bbce01f1e017e8: Status 404 returned error can't find the container with id bb8725ad925552762b8c207ef54ce8c352f73622fb8e91b557bbce01f1e017e8 Nov 03 22:41:45.589809 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.589776 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:45.589894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.589845 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:45.589894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.589875 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:45.591390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.591365 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h"] Nov 03 22:41:45.994943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.994901 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" event={"ID":"a85d19cf-d09f-427c-a04c-f97d95723d99","Type":"ContainerStarted","Data":"9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1"} Nov 03 22:41:45.995392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:45.994973 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" event={"ID":"a85d19cf-d09f-427c-a04c-f97d95723d99","Type":"ContainerStarted","Data":"bb8725ad925552762b8c207ef54ce8c352f73622fb8e91b557bbce01f1e017e8"} Nov 03 22:41:46.120167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:46.117783 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" podStartSLOduration=2.117762969 podStartE2EDuration="2.117762969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:46.114029825 +0000 UTC m=+2063.094763478" watchObservedRunningTime="2026-11-03 22:41:46.117762969 +0000 UTC m=+2063.098496624" Nov 03 22:41:46.161188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:46.161147 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:46.167459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:46.167432 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:46.307844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:46.307757 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:46.992326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:46.992244 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:41:47.003937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.003907 4298 generic.go:356] "Generic (PLEG): container finished" podID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerID="d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c" exitCode=0 Nov 03 22:41:47.004300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.003939 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" event={"ID":"68cf1bc3-4867-4446-866b-249d8cb2c7c3","Type":"ContainerDied","Data":"d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c"} Nov 03 22:41:47.004300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.003957 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" Nov 03 22:41:47.004300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.003997 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h" event={"ID":"68cf1bc3-4867-4446-866b-249d8cb2c7c3","Type":"ContainerDied","Data":"75e2a5199f691d83baf9a29ef7539231a8225bbb8c4b9c57fe1752d89defbcbb"} Nov 03 22:41:47.004300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.004026 4298 scope.go:122] "RemoveContainer" containerID="d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c" Nov 03 22:41:47.004487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.004406 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:47.005356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.005341 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:41:47.013944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.013923 4298 scope.go:122] "RemoveContainer" containerID="0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef" Nov 03 22:41:47.023246 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.023228 4298 scope.go:122] "RemoveContainer" containerID="d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c" Nov 03 22:41:47.023580 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:41:47.023561 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c\": container with ID starting with d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c not found: ID does not exist" containerID="d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c" Nov 03 22:41:47.023632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.023591 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c"} err="failed to get container status \"d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c\": rpc error: code = NotFound desc = could not find container \"d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c\": container with ID starting with d6e9f8e9774199d3974f362a1a8e01f1ba0230c25bf07a4e8670560d04d7218c not found: ID does not exist" Nov 03 22:41:47.023632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.023609 4298 scope.go:122] "RemoveContainer" containerID="0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef" Nov 03 22:41:47.023916 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:41:47.023897 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef\": container with ID starting with 0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef not found: ID does not exist" containerID="0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef" Nov 03 22:41:47.023980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.023924 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef"} err="failed to get container status \"0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef\": rpc error: code = NotFound desc = could not find container \"0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef\": container with ID starting with 0ab3f9b6134fba3b29c3adfab7617560ae28e2bdea4cc08f89f796217588ebef not found: ID does not exist" Nov 03 22:41:47.060231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.059062 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-socket\") pod \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " Nov 03 22:41:47.060231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.059440 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-envoy\") pod \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " Nov 03 22:41:47.060231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.059515 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istiod-ca-cert\") pod \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " Nov 03 22:41:47.060231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.059595 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-data\") pod \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " Nov 03 22:41:47.060231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.059677 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-certs\") pod \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " Nov 03 22:41:47.060231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.059721 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-podinfo\") pod \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " Nov 03 22:41:47.060231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.059845 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-credential-socket\") pod \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " Nov 03 22:41:47.060231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.059912 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-token\" (UniqueName: \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-token\") pod \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " Nov 03 22:41:47.060231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.059968 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-kube-api-access-gztx6\" (UniqueName: \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-kube-api-access-gztx6\") pod \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\" (UID: \"68cf1bc3-4867-4446-866b-249d8cb2c7c3\") " Nov 03 22:41:47.061265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.060753 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-socket" pod "68cf1bc3-4867-4446-866b-249d8cb2c7c3" (UID: "68cf1bc3-4867-4446-866b-249d8cb2c7c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:47.061265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.061105 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:47.061837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.061575 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-credential-socket" pod "68cf1bc3-4867-4446-866b-249d8cb2c7c3" (UID: "68cf1bc3-4867-4446-866b-249d8cb2c7c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:47.062008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.061865 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-certs" pod "68cf1bc3-4867-4446-866b-249d8cb2c7c3" (UID: "68cf1bc3-4867-4446-866b-249d8cb2c7c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:47.062008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.061921 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-data" pod "68cf1bc3-4867-4446-866b-249d8cb2c7c3" (UID: "68cf1bc3-4867-4446-866b-249d8cb2c7c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:47.065168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.063753 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istiod-ca-cert" pod "68cf1bc3-4867-4446-866b-249d8cb2c7c3" (UID: "68cf1bc3-4867-4446-866b-249d8cb2c7c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:47.065168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.064863 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-token" pod "68cf1bc3-4867-4446-866b-249d8cb2c7c3" (UID: "68cf1bc3-4867-4446-866b-249d8cb2c7c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:47.065330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.065163 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-envoy" pod "68cf1bc3-4867-4446-866b-249d8cb2c7c3" (UID: "68cf1bc3-4867-4446-866b-249d8cb2c7c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:47.066560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.066439 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-kube-api-access-gztx6" pod "68cf1bc3-4867-4446-866b-249d8cb2c7c3" (UID: "68cf1bc3-4867-4446-866b-249d8cb2c7c3"). InnerVolumeSpecName "kube-api-access-gztx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:47.066560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.066491 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-podinfo" pod "68cf1bc3-4867-4446-866b-249d8cb2c7c3" (UID: "68cf1bc3-4867-4446-866b-249d8cb2c7c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:41:47.163360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.163320 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:47.163360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.163359 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:47.163615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.163373 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:47.163615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.163387 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:47.163615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.163403 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:47.163615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.163418 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cf1bc3-4867-4446-866b-249d8cb2c7c3-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:47.163615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.163435 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:47.163615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.163449 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gztx6\" (UniqueName: \"kubernetes.io/projected/68cf1bc3-4867-4446-866b-249d8cb2c7c3-kube-api-access-gztx6\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:47.481596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.481558 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h"] Nov 03 22:41:47.492673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:47.488099 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-094053-openshift-default-7f79f57cdf-bz77h"] Nov 03 22:41:47.554919 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:41:47.554881 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:41:48.082734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:48.082695 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:48.355084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:48.355001 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:48.539199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:48.539164 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:48.545004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:48.544972 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp"] Nov 03 22:41:48.586906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:48.586870 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f61dc0-lw6lp"] Nov 03 22:41:48.963288 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:48.963254 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:49.136118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:49.136091 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" path="/var/lib/kubelet/pods/68cf1bc3-4867-4446-866b-249d8cb2c7c3/volumes" Nov 03 22:41:49.136581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:49.136567 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1d2964-f777-435a-aeef-c2e2fe843712" path="/var/lib/kubelet/pods/ab1d2964-f777-435a-aeef-c2e2fe843712/volumes" Nov 03 22:41:50.145588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:50.145559 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:50.345839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:50.345803 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:50.521396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:50.521356 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:53.457479 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:53.457445 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:53.657161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:53.657124 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:53.861847 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:53.861749 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:54.008669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.008033 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8"] Nov 03 22:41:54.012665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.008924 4298 cpu_manager.go:414] "containerMap: removing container" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" Nov 03 22:41:54.012665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.008947 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" Nov 03 22:41:54.012665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.009152 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" Nov 03 22:41:54.012665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.009167 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" Nov 03 22:41:54.034680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.034198 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:41:54.105659 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.105603 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f881e3\"" Nov 03 22:41:54.106064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.106045 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f881e3-dockercfg-tl9gg\"" Nov 03 22:41:54.120430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.120340 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8"] Nov 03 22:41:54.166017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.165980 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt"] Nov 03 22:41:54.166994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.166860 4298 cpu_manager.go:414] "containerMap: removing container" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" Nov 03 22:41:54.166994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.166880 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68cf1bc3-4867-4446-866b-249d8cb2c7c3" containerName="istio-proxy" Nov 03 22:41:54.191328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.191298 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:41:54.219056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.219026 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8\" (UID: \"bb6eb1e2-b676-460c-85cc-a82b1f524cb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:41:54.219173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.219122 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hlt\" (UniqueName: \"kubernetes.io/projected/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-kube-api-access-m2hlt\") pod \"payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8\" (UID: \"bb6eb1e2-b676-460c-85cc-a82b1f524cb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:41:54.219257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.219234 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vgx4\" (UniqueName: \"kubernetes.io/projected/2741d23e-17ab-4c6a-8d88-1250c1a3db92-kube-api-access-8vgx4\") pod \"payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt\" (UID: \"2741d23e-17ab-4c6a-8d88-1250c1a3db92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:41:54.219391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.219375 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741d23e-17ab-4c6a-8d88-1250c1a3db92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt\" (UID: \"2741d23e-17ab-4c6a-8d88-1250c1a3db92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:41:54.227745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.227718 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt"] Nov 03 22:41:54.320087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.320048 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vgx4\" (UniqueName: \"kubernetes.io/projected/2741d23e-17ab-4c6a-8d88-1250c1a3db92-kube-api-access-8vgx4\") pod \"payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt\" (UID: \"2741d23e-17ab-4c6a-8d88-1250c1a3db92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:41:54.320237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.320158 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741d23e-17ab-4c6a-8d88-1250c1a3db92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt\" (UID: \"2741d23e-17ab-4c6a-8d88-1250c1a3db92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:41:54.320237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.320200 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8\" (UID: \"bb6eb1e2-b676-460c-85cc-a82b1f524cb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:41:54.320303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.320286 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hlt\" (UniqueName: \"kubernetes.io/projected/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-kube-api-access-m2hlt\") pod \"payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8\" (UID: \"bb6eb1e2-b676-460c-85cc-a82b1f524cb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:41:54.321156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.321133 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741d23e-17ab-4c6a-8d88-1250c1a3db92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt\" (UID: \"2741d23e-17ab-4c6a-8d88-1250c1a3db92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:41:54.321199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.321133 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8\" (UID: \"bb6eb1e2-b676-460c-85cc-a82b1f524cb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:41:54.377501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.377432 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vgx4\" (UniqueName: \"kubernetes.io/projected/2741d23e-17ab-4c6a-8d88-1250c1a3db92-kube-api-access-8vgx4\") pod \"payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt\" (UID: \"2741d23e-17ab-4c6a-8d88-1250c1a3db92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:41:54.378099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.378073 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hlt\" (UniqueName: \"kubernetes.io/projected/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-kube-api-access-m2hlt\") pod \"payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8\" (UID: \"bb6eb1e2-b676-460c-85cc-a82b1f524cb5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:41:54.506234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.506196 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:41:54.676680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.676558 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:41:54.877383 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:41:54.877348 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2741d23e_17ab_4c6a_8d88_1250c1a3db92.slice/crio-47308229684e3b2943ecd7a44525904247453886caef07a90ad211405e917fca WatchSource:0}: Error finding container 47308229684e3b2943ecd7a44525904247453886caef07a90ad211405e917fca: Status 404 returned error can't find the container with id 47308229684e3b2943ecd7a44525904247453886caef07a90ad211405e917fca Nov 03 22:41:54.881234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:54.881214 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt"] Nov 03 22:41:55.051483 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:41:55.051452 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6eb1e2_b676_460c_85cc_a82b1f524cb5.slice/crio-9ceef41826eafc28f4c759e252800aa6565e25673d7331b6ddfb2da2a1efaf8d WatchSource:0}: Error finding container 9ceef41826eafc28f4c759e252800aa6565e25673d7331b6ddfb2da2a1efaf8d: Status 404 returned error can't find the container with id 9ceef41826eafc28f4c759e252800aa6565e25673d7331b6ddfb2da2a1efaf8d Nov 03 22:41:55.055721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.055695 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8"] Nov 03 22:41:55.113965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.113932 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" event={"ID":"2741d23e-17ab-4c6a-8d88-1250c1a3db92","Type":"ContainerStarted","Data":"7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b"} Nov 03 22:41:55.114099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.113992 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" event={"ID":"2741d23e-17ab-4c6a-8d88-1250c1a3db92","Type":"ContainerStarted","Data":"47308229684e3b2943ecd7a44525904247453886caef07a90ad211405e917fca"} Nov 03 22:41:55.114099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.114024 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:41:55.115314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.115283 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" event={"ID":"bb6eb1e2-b676-460c-85cc-a82b1f524cb5","Type":"ContainerStarted","Data":"9ceef41826eafc28f4c759e252800aa6565e25673d7331b6ddfb2da2a1efaf8d"} Nov 03 22:41:55.155436 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.155408 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:55.232469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.232383 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" podStartSLOduration=1.232362712 podStartE2EDuration="1.232362712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:55.229104759 +0000 UTC m=+2072.209838412" watchObservedRunningTime="2026-11-03 22:41:55.232362712 +0000 UTC m=+2072.213096365" Nov 03 22:41:55.357102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.357058 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:55.557724 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.548273 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:55.637759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.636823 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn"] Nov 03 22:41:55.664328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.664292 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.668423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.667741 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq"] Nov 03 22:41:55.672523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.672499 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.675429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.675392 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn"] Nov 03 22:41:55.719326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.719262 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq"] Nov 03 22:41:55.765043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765006 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.765241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765059 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.765241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765155 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.765241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765193 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.765403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765331 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.765403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765362 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.765477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765424 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kt86\" (UniqueName: \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-kube-api-access-9kt86\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.765477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765472 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.765554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765498 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.765554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765539 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85qvt\" (UniqueName: \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-kube-api-access-85qvt\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.765672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765564 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c11a285-70b0-4657-9733-696a97ceabb9-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.765736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765676 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c11a285-70b0-4657-9733-696a97ceabb9-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.765736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765716 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.765846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765788 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165c2233-5f8d-426f-a423-de23cb1878ed-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.765903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765852 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.765960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765931 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165c2233-5f8d-426f-a423-de23cb1878ed-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.766016 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.765984 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.766073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.766020 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.808147 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.808061 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-2a6801-openshift-default-dockercfg-nqtbg\"" Nov 03 22:41:55.866907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.866872 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165c2233-5f8d-426f-a423-de23cb1878ed-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.867110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.866921 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.867110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.866974 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165c2233-5f8d-426f-a423-de23cb1878ed-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.867110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867011 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.867110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867046 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.867348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867128 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.867348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867156 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.867348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867183 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.867348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867211 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.867348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867310 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.867348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867338 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.867721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867371 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kt86\" (UniqueName: \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-kube-api-access-9kt86\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.867721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867429 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.867721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867458 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.867721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867482 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85qvt\" (UniqueName: \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-kube-api-access-85qvt\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.867721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867491 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.867721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867516 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c11a285-70b0-4657-9733-696a97ceabb9-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.867721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867557 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c11a285-70b0-4657-9733-696a97ceabb9-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.867721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867597 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.868225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867770 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.868225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867429 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.868225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.867951 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.868392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.868246 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.868601 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.868578 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c11a285-70b0-4657-9733-696a97ceabb9-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.868712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.868627 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165c2233-5f8d-426f-a423-de23cb1878ed-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.869098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.869040 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.869222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.869199 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.869557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.869358 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.869945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.869923 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165c2233-5f8d-426f-a423-de23cb1878ed-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.870033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.869967 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.870138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.870117 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.870849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.870831 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c11a285-70b0-4657-9733-696a97ceabb9-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.930430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.930400 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.930430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.930426 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.930600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.930453 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kt86\" (UniqueName: \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-kube-api-access-9kt86\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.930699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.930631 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85qvt\" (UniqueName: \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-kube-api-access-85qvt\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:55.982895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.982864 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:55.995247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:55.994205 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:56.132210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.132173 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" event={"ID":"bb6eb1e2-b676-460c-85cc-a82b1f524cb5","Type":"ContainerStarted","Data":"3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50"} Nov 03 22:41:56.132386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.132258 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:41:56.248692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.248626 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" podStartSLOduration=3.248612131 podStartE2EDuration="3.248612131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:56.245519239 +0000 UTC m=+2073.226252892" watchObservedRunningTime="2026-11-03 22:41:56.248612131 +0000 UTC m=+2073.229345784" Nov 03 22:41:56.283141 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.283111 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt"] Nov 03 22:41:56.288632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.288607 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:56.341684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.341392 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:41:56.341684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.341499 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:41:56.344930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.344903 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt"] Nov 03 22:41:56.363862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.363826 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:56.375726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.375693 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:56.376020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.375977 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:56.376198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.376082 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbdr\" (UniqueName: \"kubernetes.io/projected/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-kube-api-access-tgbdr\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:56.399522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.396836 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn"] Nov 03 22:41:56.399522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.398419 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:56.399522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.398518 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:56.399522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.398565 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:56.420952 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:41:56.420906 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c11a285_70b0_4657_9733_696a97ceabb9.slice/crio-64b12eaeca746318c0a0e6ff049dde886c9fe8bca079e26ecbcf07da6ee63676 WatchSource:0}: Error finding container 64b12eaeca746318c0a0e6ff049dde886c9fe8bca079e26ecbcf07da6ee63676: Status 404 returned error can't find the container with id 64b12eaeca746318c0a0e6ff049dde886c9fe8bca079e26ecbcf07da6ee63676 Nov 03 22:41:56.423581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.423557 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq"] Nov 03 22:41:56.424703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.424623 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:56.424806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.424757 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:56.424879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.424801 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:41:56.477182 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.477149 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:56.477370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.477219 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbdr\" (UniqueName: \"kubernetes.io/projected/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-kube-api-access-tgbdr\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:56.477370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.477313 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:56.477934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.477913 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:56.479546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.479522 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:56.538564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.538539 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbdr\" (UniqueName: \"kubernetes.io/projected/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-kube-api-access-tgbdr\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:56.539961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.539906 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:56.603122 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.602850 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:56.987923 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:41:56.987895 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fd8e36_d988_43bb_a46f_028f2f9a5bd5.slice/crio-5ef5bd1954c8317d2f39a0d36dfd9d883a9f7133eb54f7a4aea064e5139ba351 WatchSource:0}: Error finding container 5ef5bd1954c8317d2f39a0d36dfd9d883a9f7133eb54f7a4aea064e5139ba351: Status 404 returned error can't find the container with id 5ef5bd1954c8317d2f39a0d36dfd9d883a9f7133eb54f7a4aea064e5139ba351 Nov 03 22:41:56.992299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:56.992272 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt"] Nov 03 22:41:57.138798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.138766 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" event={"ID":"46fd8e36-d988-43bb-a46f-028f2f9a5bd5","Type":"ContainerStarted","Data":"5ef5bd1954c8317d2f39a0d36dfd9d883a9f7133eb54f7a4aea064e5139ba351"} Nov 03 22:41:57.140269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.140250 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" event={"ID":"0c11a285-70b0-4657-9733-696a97ceabb9","Type":"ContainerStarted","Data":"32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc"} Nov 03 22:41:57.140362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.140294 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" event={"ID":"0c11a285-70b0-4657-9733-696a97ceabb9","Type":"ContainerStarted","Data":"64b12eaeca746318c0a0e6ff049dde886c9fe8bca079e26ecbcf07da6ee63676"} Nov 03 22:41:57.141849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.141824 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" event={"ID":"165c2233-5f8d-426f-a423-de23cb1878ed","Type":"ContainerStarted","Data":"d6147b5e0fd745ab4f819c62b8f01ed758b2fc26ded146f40da5438e9d25f9cc"} Nov 03 22:41:57.141989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.141864 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" event={"ID":"165c2233-5f8d-426f-a423-de23cb1878ed","Type":"ContainerStarted","Data":"1bbc85238a9f5bf054b3eec4ea1a20fcb478cc4a83a8ca96d1a637cdc733f530"} Nov 03 22:41:57.268908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.267258 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" podStartSLOduration=2.267233663 podStartE2EDuration="2.267233663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:57.263396773 +0000 UTC m=+2074.244130426" watchObservedRunningTime="2026-11-03 22:41:57.267233663 +0000 UTC m=+2074.247967318" Nov 03 22:41:57.387370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.387313 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" podStartSLOduration=2.387299626 podStartE2EDuration="2.387299626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:57.383313355 +0000 UTC m=+2074.364047009" watchObservedRunningTime="2026-11-03 22:41:57.387299626 +0000 UTC m=+2074.368033310" Nov 03 22:41:57.983033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.983004 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:41:57.985453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.985430 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 22:41:57.985614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.985593 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" podUID="165c2233-5f8d-426f-a423-de23cb1878ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 22:41:57.995408 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.995386 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:41:57.996955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.996933 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:41:57.997045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:57.996972 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" podUID="0c11a285-70b0-4657-9733-696a97ceabb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:41:58.184860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:58.184824 4298 generic.go:356] "Generic (PLEG): container finished" podID="46fd8e36-d988-43bb-a46f-028f2f9a5bd5" containerID="8b0ccd5ca8d9e32e28e87d54e2d6beb149134e8517a6754f682465e09da9f073" exitCode=0 Nov 03 22:41:58.185027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:58.184906 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" event={"ID":"46fd8e36-d988-43bb-a46f-028f2f9a5bd5","Type":"ContainerDied","Data":"8b0ccd5ca8d9e32e28e87d54e2d6beb149134e8517a6754f682465e09da9f073"} Nov 03 22:41:58.795784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:58.794974 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:58.983720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:58.983676 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 22:41:58.984159 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:58.983746 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" podUID="165c2233-5f8d-426f-a423-de23cb1878ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 22:41:58.995787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:58.995751 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:41:58.995935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:58.995809 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" podUID="0c11a285-70b0-4657-9733-696a97ceabb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:41:59.502991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.502964 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:41:59.581721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.581682 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-xks-ca\") pod \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " Nov 03 22:41:59.581721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.581730 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-kube-api-access-tgbdr\" (UniqueName: \"kubernetes.io/projected/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-kube-api-access-tgbdr\") pod \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " Nov 03 22:41:59.581938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.581802 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-openshift-ca\") pod \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\" (UID: \"46fd8e36-d988-43bb-a46f-028f2f9a5bd5\") " Nov 03 22:41:59.582536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.582514 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-openshift-ca" pod "46fd8e36-d988-43bb-a46f-028f2f9a5bd5" (UID: "46fd8e36-d988-43bb-a46f-028f2f9a5bd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:59.584057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.584029 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-xks-ca" pod "46fd8e36-d988-43bb-a46f-028f2f9a5bd5" (UID: "46fd8e36-d988-43bb-a46f-028f2f9a5bd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:59.584155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.584128 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-kube-api-access-tgbdr" pod "46fd8e36-d988-43bb-a46f-028f2f9a5bd5" (UID: "46fd8e36-d988-43bb-a46f-028f2f9a5bd5"). InnerVolumeSpecName "kube-api-access-tgbdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:59.684212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.684161 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:59.684212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.684205 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgbdr\" (UniqueName: \"kubernetes.io/projected/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-kube-api-access-tgbdr\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:59.684212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.684220 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fd8e36-d988-43bb-a46f-028f2f9a5bd5-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:59.983600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.983458 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 22:41:59.983600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.983511 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" podUID="165c2233-5f8d-426f-a423-de23cb1878ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 22:41:59.996220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.996189 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:41:59.996377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:41:59.996250 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" podUID="0c11a285-70b0-4657-9733-696a97ceabb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:42:00.201834 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:00.201799 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" event={"ID":"46fd8e36-d988-43bb-a46f-028f2f9a5bd5","Type":"ContainerDied","Data":"5ef5bd1954c8317d2f39a0d36dfd9d883a9f7133eb54f7a4aea064e5139ba351"} Nov 03 22:42:00.201994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:00.201854 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ef5bd1954c8317d2f39a0d36dfd9d883a9f7133eb54f7a4aea064e5139ba351" Nov 03 22:42:00.201994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:00.201910 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt" Nov 03 22:42:00.984157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:00.984122 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 22:42:00.984587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:00.984181 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" podUID="165c2233-5f8d-426f-a423-de23cb1878ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 22:42:00.995788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:00.995755 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:42:00.995870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:00.995808 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" podUID="0c11a285-70b0-4657-9733-696a97ceabb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:42:01.983964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:01.983921 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 22:42:01.984190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:01.983987 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" podUID="165c2233-5f8d-426f-a423-de23cb1878ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 22:42:01.996252 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:01.996210 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:42:01.996447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:01.996299 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" podUID="0c11a285-70b0-4657-9733-696a97ceabb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:42:03.001153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:03.001116 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:42:03.001613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:03.001518 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:42:03.002367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:03.002341 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:42:03.027675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:03.025275 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:42:03.027675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:03.025897 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:42:03.027675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:03.026849 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:42:03.751733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:03.751689 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:06.135327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:06.135297 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:42:06.878801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:06.878766 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm"] Nov 03 22:42:06.879403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:06.879390 4298 cpu_manager.go:414] "containerMap: removing container" podUID="46fd8e36-d988-43bb-a46f-028f2f9a5bd5" containerName="revoke-keys" Nov 03 22:42:06.879448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:06.879403 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fd8e36-d988-43bb-a46f-028f2f9a5bd5" containerName="revoke-keys" Nov 03 22:42:06.879549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:06.879540 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fd8e36-d988-43bb-a46f-028f2f9a5bd5" containerName="revoke-keys" Nov 03 22:42:06.883610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:06.883593 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:06.936256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:06.936225 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bc88eb47-openshift-default-dockercfg-h27hg\"" Nov 03 22:42:06.991854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:06.991822 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm"] Nov 03 22:42:07.009343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.009310 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c39859-96fe-412f-8cc9-63b7c5f29bea-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.009507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.009373 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.009507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.009485 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpvsg\" (UniqueName: \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-kube-api-access-mpvsg\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.009621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.009593 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.009671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.009656 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.009705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.009681 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.009788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.009776 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.009827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.009810 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.009862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.009828 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.111583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.111540 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.111767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.111597 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.111767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.111621 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.111767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.111695 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.111767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.111741 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.111973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.111769 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.111973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.111888 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c39859-96fe-412f-8cc9-63b7c5f29bea-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.111973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.111962 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.112118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.112027 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvsg\" (UniqueName: \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-kube-api-access-mpvsg\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.112118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.112051 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.112118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.112087 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.112446 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.112423 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.112546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.112433 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.112799 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.112772 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c39859-96fe-412f-8cc9-63b7c5f29bea-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.114135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.114113 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.114418 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.114403 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.169418 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.169332 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.169800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.169461 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpvsg\" (UniqueName: \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-kube-api-access-mpvsg\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.198316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.198281 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:07.623318 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:42:07.623285 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c39859_96fe_412f_8cc9_63b7c5f29bea.slice/crio-0b45ee90090f3ac053541c2d06af1f4d12425a014bbd63710fd5309589bf6820 WatchSource:0}: Error finding container 0b45ee90090f3ac053541c2d06af1f4d12425a014bbd63710fd5309589bf6820: Status 404 returned error can't find the container with id 0b45ee90090f3ac053541c2d06af1f4d12425a014bbd63710fd5309589bf6820 Nov 03 22:42:07.626708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.626671 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:07.626825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.626739 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:07.626825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.626769 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:07.628291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.628265 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm"] Nov 03 22:42:07.792292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.792258 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7"] Nov 03 22:42:07.905144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:07.905045 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq"] Nov 03 22:42:08.188905 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:08.188820 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:42:08.193867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:08.193842 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:08.288479 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:08.288444 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" event={"ID":"52c39859-96fe-412f-8cc9-63b7c5f29bea","Type":"ContainerStarted","Data":"4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8"} Nov 03 22:42:08.288677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:08.288524 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" event={"ID":"52c39859-96fe-412f-8cc9-63b7c5f29bea","Type":"ContainerStarted","Data":"0b45ee90090f3ac053541c2d06af1f4d12425a014bbd63710fd5309589bf6820"} Nov 03 22:42:08.399483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:08.399445 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:08.466764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:08.466710 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" podStartSLOduration=2.466693273 podStartE2EDuration="2.466693273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:08.461214418 +0000 UTC m=+2085.441948084" watchObservedRunningTime="2026-11-03 22:42:08.466693273 +0000 UTC m=+2085.447426926" Nov 03 22:42:08.714020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:08.713964 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:08.953842 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:08.953808 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:09.199303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:09.199270 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:09.201117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:09.201089 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:42:09.201248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:09.201151 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" podUID="52c39859-96fe-412f-8cc9-63b7c5f29bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 22:42:10.199562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:10.199521 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:42:10.200099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:10.199589 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" podUID="52c39859-96fe-412f-8cc9-63b7c5f29bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 22:42:11.199280 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:11.199240 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:42:11.199607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:11.199305 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" podUID="52c39859-96fe-412f-8cc9-63b7c5f29bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 22:42:12.198876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:12.198841 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:42:12.199066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:12.198894 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" podUID="52c39859-96fe-412f-8cc9-63b7c5f29bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 22:42:12.421009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:12.420979 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww"] Nov 03 22:42:12.421411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:12.421154 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" containerID="cri-o://890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a" gracePeriod=30 Nov 03 22:42:12.473283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:12.473174 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml"] Nov 03 22:42:12.473806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:12.473761 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" containerID="cri-o://666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d" gracePeriod=30 Nov 03 22:42:12.792993 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:12.792901 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" podUID="8f607014-d55b-45d4-943a-f1a662f9edfd" containerName="payload-processing" containerID="cri-o://ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe" gracePeriod=25 Nov 03 22:42:12.887585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:12.887502 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:12.906181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:12.906149 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" podUID="c5bc5843-6681-4320-ae7f-0b8eda6e2961" containerName="payload-pre-processing" containerID="cri-o://25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937" gracePeriod=25 Nov 03 22:42:13.164594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.164563 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:42:13.198841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.198785 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:42:13.199933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.198843 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" podUID="52c39859-96fe-412f-8cc9-63b7c5f29bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 22:42:13.272889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.272861 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:42:13.274158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.274140 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f607014-d55b-45d4-943a-f1a662f9edfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f607014-d55b-45d4-943a-f1a662f9edfd-plugins-config-volume\") pod \"8f607014-d55b-45d4-943a-f1a662f9edfd\" (UID: \"8f607014-d55b-45d4-943a-f1a662f9edfd\") " Nov 03 22:42:13.274244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.274231 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f607014-d55b-45d4-943a-f1a662f9edfd-kube-api-access-kgvw6\" (UniqueName: \"kubernetes.io/projected/8f607014-d55b-45d4-943a-f1a662f9edfd-kube-api-access-kgvw6\") pod \"8f607014-d55b-45d4-943a-f1a662f9edfd\" (UID: \"8f607014-d55b-45d4-943a-f1a662f9edfd\") " Nov 03 22:42:13.274606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.274587 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f607014-d55b-45d4-943a-f1a662f9edfd-plugins-config-volume" pod "8f607014-d55b-45d4-943a-f1a662f9edfd" (UID: "8f607014-d55b-45d4-943a-f1a662f9edfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:13.275136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.275120 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f607014-d55b-45d4-943a-f1a662f9edfd-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:13.276784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.276762 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f607014-d55b-45d4-943a-f1a662f9edfd-kube-api-access-kgvw6" pod "8f607014-d55b-45d4-943a-f1a662f9edfd" (UID: "8f607014-d55b-45d4-943a-f1a662f9edfd"). InnerVolumeSpecName "kube-api-access-kgvw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:13.370168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.370090 4298 generic.go:356] "Generic (PLEG): container finished" podID="8f607014-d55b-45d4-943a-f1a662f9edfd" containerID="ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe" exitCode=0 Nov 03 22:42:13.370551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.370245 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" event={"ID":"8f607014-d55b-45d4-943a-f1a662f9edfd","Type":"ContainerDied","Data":"ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe"} Nov 03 22:42:13.370735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.370303 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" Nov 03 22:42:13.371016 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.370860 4298 scope.go:122] "RemoveContainer" containerID="ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe" Nov 03 22:42:13.371132 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.371111 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7" event={"ID":"8f607014-d55b-45d4-943a-f1a662f9edfd","Type":"ContainerDied","Data":"82c2338bf9bf32350943135dc78fac2eb7f14ecb8c4134c2e44835aa7a846a31"} Nov 03 22:42:13.372445 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.372421 4298 generic.go:356] "Generic (PLEG): container finished" podID="c5bc5843-6681-4320-ae7f-0b8eda6e2961" containerID="25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937" exitCode=0 Nov 03 22:42:13.372516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.372463 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" Nov 03 22:42:13.372554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.372462 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" event={"ID":"c5bc5843-6681-4320-ae7f-0b8eda6e2961","Type":"ContainerDied","Data":"25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937"} Nov 03 22:42:13.372605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.372588 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq" event={"ID":"c5bc5843-6681-4320-ae7f-0b8eda6e2961","Type":"ContainerDied","Data":"75a893a6682b694b9115b7070808497823882455052302203b24bdd3a20b24cb"} Nov 03 22:42:13.375682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.375659 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5bc5843-6681-4320-ae7f-0b8eda6e2961-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bc5843-6681-4320-ae7f-0b8eda6e2961-plugins-config-volume\") pod \"c5bc5843-6681-4320-ae7f-0b8eda6e2961\" (UID: \"c5bc5843-6681-4320-ae7f-0b8eda6e2961\") " Nov 03 22:42:13.375943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.375909 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5bc5843-6681-4320-ae7f-0b8eda6e2961-kube-api-access-5xnkp\" (UniqueName: \"kubernetes.io/projected/c5bc5843-6681-4320-ae7f-0b8eda6e2961-kube-api-access-5xnkp\") pod \"c5bc5843-6681-4320-ae7f-0b8eda6e2961\" (UID: \"c5bc5843-6681-4320-ae7f-0b8eda6e2961\") " Nov 03 22:42:13.375943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.375931 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5bc5843-6681-4320-ae7f-0b8eda6e2961-plugins-config-volume" pod "c5bc5843-6681-4320-ae7f-0b8eda6e2961" (UID: "c5bc5843-6681-4320-ae7f-0b8eda6e2961"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:13.376571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.376548 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bc5843-6681-4320-ae7f-0b8eda6e2961-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:13.376655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.376578 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgvw6\" (UniqueName: \"kubernetes.io/projected/8f607014-d55b-45d4-943a-f1a662f9edfd-kube-api-access-kgvw6\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:13.379209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.378788 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5bc5843-6681-4320-ae7f-0b8eda6e2961-kube-api-access-5xnkp" pod "c5bc5843-6681-4320-ae7f-0b8eda6e2961" (UID: "c5bc5843-6681-4320-ae7f-0b8eda6e2961"). InnerVolumeSpecName "kube-api-access-5xnkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:13.382555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.382540 4298 scope.go:122] "RemoveContainer" containerID="ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe" Nov 03 22:42:13.382885 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:42:13.382867 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe\": container with ID starting with ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe not found: ID does not exist" containerID="ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe" Nov 03 22:42:13.382929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.382894 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe"} err="failed to get container status \"ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe\": rpc error: code = NotFound desc = could not find container \"ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe\": container with ID starting with ac219b5720d9c1d05f603813ef0582a4e542fff3b50a45bdbedd8cb96e7f0dfe not found: ID does not exist" Nov 03 22:42:13.382929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.382909 4298 scope.go:122] "RemoveContainer" containerID="25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937" Nov 03 22:42:13.392851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.392830 4298 scope.go:122] "RemoveContainer" containerID="25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937" Nov 03 22:42:13.393122 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:42:13.393101 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937\": container with ID starting with 25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937 not found: ID does not exist" containerID="25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937" Nov 03 22:42:13.393198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.393131 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937"} err="failed to get container status \"25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937\": rpc error: code = NotFound desc = could not find container \"25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937\": container with ID starting with 25152c6e6d7fffd9bc206f5e6a0250457e7c261b87059eca1e6bb0a207aee937 not found: ID does not exist" Nov 03 22:42:13.478367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.478329 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xnkp\" (UniqueName: \"kubernetes.io/projected/c5bc5843-6681-4320-ae7f-0b8eda6e2961-kube-api-access-5xnkp\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:13.554462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.554425 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7"] Nov 03 22:42:13.559149 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.559124 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e7d36a-889f8748d-2p5k7"] Nov 03 22:42:13.796037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.795996 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq"] Nov 03 22:42:13.800931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:13.800901 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7d36a-5946d8459-9w6zq"] Nov 03 22:42:14.251699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:14.251638 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:14.252267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:14.252244 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:14.254075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:14.254055 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:42:14.406337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:14.406303 4298 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:42:14.406507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:14.406362 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:42:15.135893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:15.135861 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:15.136378 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:15.136349 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f607014-d55b-45d4-943a-f1a662f9edfd" path="/var/lib/kubelet/pods/8f607014-d55b-45d4-943a-f1a662f9edfd/volumes" Nov 03 22:42:15.136885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:15.136870 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5bc5843-6681-4320-ae7f-0b8eda6e2961" path="/var/lib/kubelet/pods/c5bc5843-6681-4320-ae7f-0b8eda6e2961/volumes" Nov 03 22:42:16.880404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:16.876169 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:17.561755 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:42:17.561723 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:42:17.763622 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.763467 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:42:17.866225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.866186 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-data\") pod \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " Nov 03 22:42:17.866395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.866322 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-kube-api-access-d7grm\" (UniqueName: \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-kube-api-access-d7grm\") pod \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " Nov 03 22:42:17.866456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.866443 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-certs\") pod \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " Nov 03 22:42:17.866515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.866496 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-envoy\") pod \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " Nov 03 22:42:17.866627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.866537 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-token\" (UniqueName: \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-token\") pod \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " Nov 03 22:42:17.866712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.866678 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-credential-socket\") pod \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " Nov 03 22:42:17.866801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.866783 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-socket\") pod \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " Nov 03 22:42:17.866862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.866844 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istiod-ca-cert\") pod \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " Nov 03 22:42:17.866916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.866891 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-podinfo\") pod \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\" (UID: \"98055b9b-6d24-4b65-8e53-d0279f4f6fb0\") " Nov 03 22:42:17.869373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.869201 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-data" pod "98055b9b-6d24-4b65-8e53-d0279f4f6fb0" (UID: "98055b9b-6d24-4b65-8e53-d0279f4f6fb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.869626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.869598 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-socket" pod "98055b9b-6d24-4b65-8e53-d0279f4f6fb0" (UID: "98055b9b-6d24-4b65-8e53-d0279f4f6fb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.869848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.869794 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istiod-ca-cert" pod "98055b9b-6d24-4b65-8e53-d0279f4f6fb0" (UID: "98055b9b-6d24-4b65-8e53-d0279f4f6fb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:17.869941 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.869891 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-certs" pod "98055b9b-6d24-4b65-8e53-d0279f4f6fb0" (UID: "98055b9b-6d24-4b65-8e53-d0279f4f6fb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.870055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.870034 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-credential-socket" pod "98055b9b-6d24-4b65-8e53-d0279f4f6fb0" (UID: "98055b9b-6d24-4b65-8e53-d0279f4f6fb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.871062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.871035 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-envoy" pod "98055b9b-6d24-4b65-8e53-d0279f4f6fb0" (UID: "98055b9b-6d24-4b65-8e53-d0279f4f6fb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.871789 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.871740 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-token" pod "98055b9b-6d24-4b65-8e53-d0279f4f6fb0" (UID: "98055b9b-6d24-4b65-8e53-d0279f4f6fb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:17.871789 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.871759 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-podinfo" pod "98055b9b-6d24-4b65-8e53-d0279f4f6fb0" (UID: "98055b9b-6d24-4b65-8e53-d0279f4f6fb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:17.872760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.872733 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-kube-api-access-d7grm" pod "98055b9b-6d24-4b65-8e53-d0279f4f6fb0" (UID: "98055b9b-6d24-4b65-8e53-d0279f4f6fb0"). InnerVolumeSpecName "kube-api-access-d7grm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:17.922040 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.922008 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:42:17.968595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.968558 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7grm\" (UniqueName: \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-kube-api-access-d7grm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.968595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.968589 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.968595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.968598 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.968595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.968606 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.968899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.968615 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.968899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.968623 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.968899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.968631 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.968899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.968691 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.968899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:17.968706 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98055b9b-6d24-4b65-8e53-d0279f4f6fb0-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.070022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.069992 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-credential-socket\") pod \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " Nov 03 22:42:18.070240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.070037 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-kube-api-access-cld42\" (UniqueName: \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-kube-api-access-cld42\") pod \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " Nov 03 22:42:18.070240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.070072 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-envoy\") pod \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " Nov 03 22:42:18.070240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.070113 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-data\") pod \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " Nov 03 22:42:18.070419 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.070319 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-credential-socket" pod "927ec6e4-c00c-4cf7-a1c6-8317f9552a95" (UID: "927ec6e4-c00c-4cf7-a1c6-8317f9552a95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:18.070660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.070406 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-socket\") pod \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " Nov 03 22:42:18.070768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.070612 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-data" pod "927ec6e4-c00c-4cf7-a1c6-8317f9552a95" (UID: "927ec6e4-c00c-4cf7-a1c6-8317f9552a95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:18.070842 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.070787 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-socket" pod "927ec6e4-c00c-4cf7-a1c6-8317f9552a95" (UID: "927ec6e4-c00c-4cf7-a1c6-8317f9552a95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:18.070904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.070881 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-token\" (UniqueName: \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-token\") pod \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " Nov 03 22:42:18.071116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.071022 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istiod-ca-cert\") pod \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " Nov 03 22:42:18.071427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.071356 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istiod-ca-cert" pod "927ec6e4-c00c-4cf7-a1c6-8317f9552a95" (UID: "927ec6e4-c00c-4cf7-a1c6-8317f9552a95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:18.071427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.071442 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-certs\") pod \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " Nov 03 22:42:18.071775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.071754 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-certs" pod "927ec6e4-c00c-4cf7-a1c6-8317f9552a95" (UID: "927ec6e4-c00c-4cf7-a1c6-8317f9552a95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:18.071867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.071826 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-podinfo\") pod \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\" (UID: \"927ec6e4-c00c-4cf7-a1c6-8317f9552a95\") " Nov 03 22:42:18.072946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.072917 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.073061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.072942 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-kube-api-access-cld42" pod "927ec6e4-c00c-4cf7-a1c6-8317f9552a95" (UID: "927ec6e4-c00c-4cf7-a1c6-8317f9552a95"). InnerVolumeSpecName "kube-api-access-cld42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:18.073061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.072948 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.073061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.072995 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.073061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.073013 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.073061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.073031 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.073326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.073300 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-envoy" pod "927ec6e4-c00c-4cf7-a1c6-8317f9552a95" (UID: "927ec6e4-c00c-4cf7-a1c6-8317f9552a95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:18.073384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.073341 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-token" pod "927ec6e4-c00c-4cf7-a1c6-8317f9552a95" (UID: "927ec6e4-c00c-4cf7-a1c6-8317f9552a95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:18.074221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.074188 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-podinfo" pod "927ec6e4-c00c-4cf7-a1c6-8317f9552a95" (UID: "927ec6e4-c00c-4cf7-a1c6-8317f9552a95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:18.174521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.174488 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.174521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.174519 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.174521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.174529 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.174801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.174538 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cld42\" (UniqueName: \"kubernetes.io/projected/927ec6e4-c00c-4cf7-a1c6-8317f9552a95-kube-api-access-cld42\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.389840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.389753 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:18.413110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.413080 4298 generic.go:356] "Generic (PLEG): container finished" podID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerID="666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d" exitCode=0 Nov 03 22:42:18.413288 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.413126 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" Nov 03 22:42:18.413288 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.413163 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" event={"ID":"927ec6e4-c00c-4cf7-a1c6-8317f9552a95","Type":"ContainerDied","Data":"666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d"} Nov 03 22:42:18.413288 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.413223 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml" event={"ID":"927ec6e4-c00c-4cf7-a1c6-8317f9552a95","Type":"ContainerDied","Data":"537ef1904a01b2bccd1a21227edb5fddd1c5a923e3c5a29e322dcb762253af86"} Nov 03 22:42:18.413288 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.413251 4298 scope.go:122] "RemoveContainer" containerID="666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d" Nov 03 22:42:18.420199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.416294 4298 generic.go:356] "Generic (PLEG): container finished" podID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerID="890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a" exitCode=0 Nov 03 22:42:18.420199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.416480 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" event={"ID":"98055b9b-6d24-4b65-8e53-d0279f4f6fb0","Type":"ContainerDied","Data":"890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a"} Nov 03 22:42:18.420199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.416525 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" event={"ID":"98055b9b-6d24-4b65-8e53-d0279f4f6fb0","Type":"ContainerDied","Data":"08ce95015c192c135f7b6428cd0a5f16ee1b43d9b53ae91857cf799c1209032f"} Nov 03 22:42:18.420199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.416605 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww" Nov 03 22:42:18.430907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.430889 4298 scope.go:122] "RemoveContainer" containerID="666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d" Nov 03 22:42:18.431209 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:42:18.431188 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d\": container with ID starting with 666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d not found: ID does not exist" containerID="666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d" Nov 03 22:42:18.431257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.431216 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d"} err="failed to get container status \"666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d\": rpc error: code = NotFound desc = could not find container \"666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d\": container with ID starting with 666dc188458d61405bb4f12bf7dba92d673e9762290e946864e1cca55b197f7d not found: ID does not exist" Nov 03 22:42:18.431257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.431234 4298 scope.go:122] "RemoveContainer" containerID="890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a" Nov 03 22:42:18.441438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.441415 4298 scope.go:122] "RemoveContainer" containerID="890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a" Nov 03 22:42:18.441729 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:42:18.441706 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a\": container with ID starting with 890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a not found: ID does not exist" containerID="890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a" Nov 03 22:42:18.441811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.441736 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a"} err="failed to get container status \"890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a\": rpc error: code = NotFound desc = could not find container \"890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a\": container with ID starting with 890034fa305b7bd0bb17619e90e0122ce95e1ee30cad593665331e428a63052a not found: ID does not exist" Nov 03 22:42:18.583004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.582971 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml"] Nov 03 22:42:18.631769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.631738 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-8t5ml"] Nov 03 22:42:18.703507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.703476 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:18.746502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.746466 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww"] Nov 03 22:42:18.753815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.753775 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-t65ww"] Nov 03 22:42:18.885424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.885377 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:18.993256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.993163 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt"] Nov 03 22:42:18.997650 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:18.997607 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7d36a-5hrlt"] Nov 03 22:42:19.137654 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:19.137602 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fd8e36-d988-43bb-a46f-028f2f9a5bd5" path="/var/lib/kubelet/pods/46fd8e36-d988-43bb-a46f-028f2f9a5bd5/volumes" Nov 03 22:42:19.138383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:19.138365 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" path="/var/lib/kubelet/pods/927ec6e4-c00c-4cf7-a1c6-8317f9552a95/volumes" Nov 03 22:42:19.138886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:19.138872 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" path="/var/lib/kubelet/pods/98055b9b-6d24-4b65-8e53-d0279f4f6fb0/volumes" Nov 03 22:42:19.462276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:19.462237 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:19.748620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:19.748539 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:21.243385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:21.243352 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:21.696828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:21.696797 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:22.038006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.037923 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw"] Nov 03 22:42:22.038914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.038892 4298 cpu_manager.go:414] "containerMap: removing container" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" Nov 03 22:42:22.038972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.038918 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" Nov 03 22:42:22.038972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.038960 4298 cpu_manager.go:414] "containerMap: removing container" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" Nov 03 22:42:22.038972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.038967 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" Nov 03 22:42:22.039077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.038979 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c5bc5843-6681-4320-ae7f-0b8eda6e2961" containerName="payload-pre-processing" Nov 03 22:42:22.039077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.038988 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5bc5843-6681-4320-ae7f-0b8eda6e2961" containerName="payload-pre-processing" Nov 03 22:42:22.039077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.039015 4298 cpu_manager.go:414] "containerMap: removing container" podUID="8f607014-d55b-45d4-943a-f1a662f9edfd" containerName="payload-processing" Nov 03 22:42:22.039077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.039022 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f607014-d55b-45d4-943a-f1a662f9edfd" containerName="payload-processing" Nov 03 22:42:22.039197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.039148 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f607014-d55b-45d4-943a-f1a662f9edfd" containerName="payload-processing" Nov 03 22:42:22.039197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.039171 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="98055b9b-6d24-4b65-8e53-d0279f4f6fb0" containerName="istio-proxy" Nov 03 22:42:22.039197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.039185 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="927ec6e4-c00c-4cf7-a1c6-8317f9552a95" containerName="istio-proxy" Nov 03 22:42:22.039285 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.039197 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5bc5843-6681-4320-ae7f-0b8eda6e2961" containerName="payload-pre-processing" Nov 03 22:42:22.044960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.044937 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.123662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.123607 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.123864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.123737 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpmh9\" (UniqueName: \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-kube-api-access-vpmh9\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.123864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.123780 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.123864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.123831 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.123864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.123851 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.124119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.123880 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.124119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.123915 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.124119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.123942 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.124119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.124004 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.165755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.165602 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw"] Nov 03 22:42:22.225810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.225772 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.226029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.225834 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.226029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.225895 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.226029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.225952 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.226029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.226019 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.226254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.226179 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.226311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.226260 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.226392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.226357 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpmh9\" (UniqueName: \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-kube-api-access-vpmh9\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.226455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.226441 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.226515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.226488 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.226581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.226558 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.227026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.227000 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.227154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.227071 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.227154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.227134 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.229025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.229006 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.229255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.229233 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.287734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.287700 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.290943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.290841 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpmh9\" (UniqueName: \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-kube-api-access-vpmh9\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.360237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.359924 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:22.432364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.432314 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:22.633666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.633560 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr"] Nov 03 22:42:22.639167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.639145 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.694751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.694715 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b30a14-openshift-default-dockercfg-mns6t\"" Nov 03 22:42:22.698953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.698924 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr"] Nov 03 22:42:22.738062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.738023 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-socket\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.738242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.738078 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5659da87-ec7e-4434-a21c-c33ac6aacd20-istiod-ca-cert\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.738242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.738151 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-data\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.738598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.738564 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tnxt\" (UniqueName: \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-kube-api-access-9tnxt\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.738800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.738618 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-podinfo\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.738800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.738738 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-token\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.738800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.738796 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-envoy\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.738969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.738903 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-certs\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.738969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.738952 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-credential-socket\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.761515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.761466 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:22.840851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.840817 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-credential-socket\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.840901 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-socket\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.840937 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5659da87-ec7e-4434-a21c-c33ac6aacd20-istiod-ca-cert\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.840961 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-data\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.841018 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tnxt\" (UniqueName: \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-kube-api-access-9tnxt\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.841041 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-podinfo\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.841102 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-token\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.841165 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-envoy\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.841233 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-certs\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.841291 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-credential-socket\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.841329 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-socket\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.841618 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5659da87-ec7e-4434-a21c-c33ac6aacd20-istiod-ca-cert\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.841758 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-certs\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.841840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.841800 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-data\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.844037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.844008 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-podinfo\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.844197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.844153 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-envoy\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.849245 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:42:22.849205 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6549bc_c506_425a_bf96_9d3ba30b9d6c.slice/crio-a793929ac7321b47166eb3b1f55e76b286a8eb53c78a0587d212e295a13bc56a WatchSource:0}: Error finding container a793929ac7321b47166eb3b1f55e76b286a8eb53c78a0587d212e295a13bc56a: Status 404 returned error can't find the container with id a793929ac7321b47166eb3b1f55e76b286a8eb53c78a0587d212e295a13bc56a Nov 03 22:42:22.852284 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.852263 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw"] Nov 03 22:42:22.852411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.852377 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:22.852487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.852469 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:22.852537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.852522 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:22.917826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.917732 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tnxt\" (UniqueName: \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-kube-api-access-9tnxt\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.926355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.926111 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-token\") pod \"e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:22.957678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:22.956913 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:23.018571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:23.017631 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:23.458727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:23.458685 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" event={"ID":"5f6549bc-c506-425a-bf96-9d3ba30b9d6c","Type":"ContainerStarted","Data":"5adf3240ec844e840104c47ed1dad3d2778a35c4584665b06231ffdec5373ac1"} Nov 03 22:42:23.459358 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:23.459333 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" event={"ID":"5f6549bc-c506-425a-bf96-9d3ba30b9d6c","Type":"ContainerStarted","Data":"a793929ac7321b47166eb3b1f55e76b286a8eb53c78a0587d212e295a13bc56a"} Nov 03 22:42:23.559229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:23.559185 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:23.559426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:23.559405 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:23.559574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:23.559558 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:23.566830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:23.566265 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr"] Nov 03 22:42:23.579819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:23.579765 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" podStartSLOduration=2.579751338 podStartE2EDuration="2.579751338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:23.578524797 +0000 UTC m=+2100.559258449" watchObservedRunningTime="2026-11-03 22:42:23.579751338 +0000 UTC m=+2100.560484994" Nov 03 22:42:23.809633 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:23.809551 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:24.361447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:24.361417 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:24.363080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:24.363056 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:42:24.363201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:24.363111 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 03 22:42:24.468525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:24.468484 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" event={"ID":"5659da87-ec7e-4434-a21c-c33ac6aacd20","Type":"ContainerStarted","Data":"cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55"} Nov 03 22:42:24.468952 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:24.468544 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" event={"ID":"5659da87-ec7e-4434-a21c-c33ac6aacd20","Type":"ContainerStarted","Data":"3c3cd91e98105d5ebb08fb34cb0de9e9b563ce042b8b7d96f808c330ff566e84"} Nov 03 22:42:24.584969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:24.584917 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" podStartSLOduration=2.584904588 podStartE2EDuration="2.584904588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:24.584259224 +0000 UTC m=+2101.564992877" watchObservedRunningTime="2026-11-03 22:42:24.584904588 +0000 UTC m=+2101.565638240" Nov 03 22:42:24.958247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:24.958208 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:24.964939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:24.964913 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:25.081920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.081874 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:25.361482 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.361386 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:42:25.361482 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.361455 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 03 22:42:25.476251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.476213 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:25.477049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.477031 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:42:25.620948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.620870 4298 scope.go:122] "RemoveContainer" containerID="d7931fe1cb02e99e1521b45d77ae091ab7dad3d4e639e406aa760d962c1d386a" Nov 03 22:42:25.658895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.658873 4298 scope.go:122] "RemoveContainer" containerID="9a50e343457ce979f165d041284c9109d0eb3b47be9a16d0b61775f584a2b25e" Nov 03 22:42:25.669944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.669904 4298 scope.go:122] "RemoveContainer" containerID="445ca4a47b562127ccf59fc2352d42e8f77203e9efc5b28129080acd56346acf" Nov 03 22:42:25.823519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.823489 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:25.865275 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.865247 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv"] Nov 03 22:42:25.871402 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.871333 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:25.985170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:25.980390 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv"] Nov 03 22:42:26.004230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.002843 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.004230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.003004 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.004230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.003102 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.004230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.003206 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4znf2\" (UniqueName: \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-kube-api-access-4znf2\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.004230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.003239 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.004230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.003347 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.004230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.003392 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.004230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.003483 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.004230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.003535 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.017705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.017635 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:26.085630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.085597 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w"] Nov 03 22:42:26.085940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.085913 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" containerID="cri-o://2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1" gracePeriod=30 Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.104513 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.104597 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.104681 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.104760 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.104812 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.104848 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.104923 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.105018 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.105040 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znf2\" (UniqueName: \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-kube-api-access-4znf2\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.105089 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.105255 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.105837 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.106069 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.106360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.106231 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.111676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.109121 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.112428 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.112404 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.162704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.162600 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.162704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.162690 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znf2\" (UniqueName: \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-kube-api-access-4znf2\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.191099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.191054 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:26.223675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.221478 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:26.376667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.364183 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:42:26.376667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.364262 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 03 22:42:26.856177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.856150 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv"] Nov 03 22:42:26.856592 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.856297 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:26.856592 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.856368 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:26.856592 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.856400 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:26.985788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:26.985758 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:27.361140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.361103 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:42:27.361371 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.361170 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 03 22:42:27.427709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.425973 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:27.498320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.498291 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" event={"ID":"bad67035-dcf3-45d0-aad3-cca0ddd7c429","Type":"ContainerStarted","Data":"c5878756fed2b1db5cc77e9d8762e1c63a3793187085ee23cd53c9976bffe977"} Nov 03 22:42:27.498502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.498338 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" event={"ID":"bad67035-dcf3-45d0-aad3-cca0ddd7c429","Type":"ContainerStarted","Data":"81e3e4a6f214545241e2c3a0990cfe214786cce191373c10168c0dbe6ee9292e"} Nov 03 22:42:27.607693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.607659 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt"] Nov 03 22:42:27.612260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.612206 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.651327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.651265 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" podStartSLOduration=2.651244769 podStartE2EDuration="2.651244769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:27.649813237 +0000 UTC m=+2104.630546892" watchObservedRunningTime="2026-11-03 22:42:27.651244769 +0000 UTC m=+2104.631978423" Nov 03 22:42:27.669562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.669531 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:27.674815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.674791 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1b2fb1-openshift-default-dockercfg-l4shk\"" Nov 03 22:42:27.739718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.737310 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt"] Nov 03 22:42:27.879514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.800996 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a535c385-80f0-49ec-870d-7d83ccce3073-istiod-ca-cert\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.879514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.801072 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mtt\" (UniqueName: \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-kube-api-access-m2mtt\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.879514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.801101 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a535c385-80f0-49ec-870d-7d83ccce3073-istio-podinfo\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.879514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.801212 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-istio-token\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.879514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.801274 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-credential-socket\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.879514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.801299 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-data\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.879514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.801332 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-socket\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.879514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.801509 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-certs\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.879514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.801537 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-envoy\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.968911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.963361 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-istio-token\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.968911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.963469 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-credential-socket\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.968911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.963524 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-data\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.968911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.963565 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-socket\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:27.968911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.963858 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-certs\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.004091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.973742 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-envoy\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.004091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.973821 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a535c385-80f0-49ec-870d-7d83ccce3073-istiod-ca-cert\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.004091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.973887 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mtt\" (UniqueName: \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-kube-api-access-m2mtt\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.004091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.973920 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a535c385-80f0-49ec-870d-7d83ccce3073-istio-podinfo\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.004091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.975102 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-certs\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.004091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.975342 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-credential-socket\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.004091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:27.999954 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-envoy\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.004091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.000255 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a535c385-80f0-49ec-870d-7d83ccce3073-istiod-ca-cert\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.004091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.001970 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-data\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.004091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.002230 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-socket\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.007839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.007788 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a535c385-80f0-49ec-870d-7d83ccce3073-istio-podinfo\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.013698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.010695 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:28.061694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.056123 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-istio-token\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.062620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.062525 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mtt\" (UniqueName: \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-kube-api-access-m2mtt\") pod \"e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.192594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.192460 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:28.193755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.193712 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:42:28.193885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.193772 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 22:42:28.214028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.213999 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:28.258667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.257940 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:28.361227 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.361191 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:42:28.361538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.361492 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 03 22:42:28.979883 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:42:28.979849 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda535c385_80f0_49ec_870d_7d83ccce3073.slice/crio-adb247037e11456b88026e435b115ee29c00110bac64f5123e2745a309b27091 WatchSource:0}: Error finding container adb247037e11456b88026e435b115ee29c00110bac64f5123e2745a309b27091: Status 404 returned error can't find the container with id adb247037e11456b88026e435b115ee29c00110bac64f5123e2745a309b27091 Nov 03 22:42:28.983296 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.983260 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt"] Nov 03 22:42:28.983421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.983283 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:28.983421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.983351 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:28.983421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:28.983379 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:29.191792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:29.191760 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:42:29.191995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:29.191972 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 22:42:29.366718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:29.362005 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:42:29.366718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:29.362061 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 03 22:42:29.366718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:29.362585 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:29.541044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:29.541003 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" event={"ID":"a535c385-80f0-49ec-870d-7d83ccce3073","Type":"ContainerStarted","Data":"989c34827f115cc66afea8a033f30be9c8c816375483c25696dec12d6b8af6d0"} Nov 03 22:42:29.541815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:29.541789 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" event={"ID":"a535c385-80f0-49ec-870d-7d83ccce3073","Type":"ContainerStarted","Data":"adb247037e11456b88026e435b115ee29c00110bac64f5123e2745a309b27091"} Nov 03 22:42:29.697782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:29.697704 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" podStartSLOduration=2.6976817459999998 podStartE2EDuration="2.697681746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:29.672024454 +0000 UTC m=+2106.652758108" watchObservedRunningTime="2026-11-03 22:42:29.697681746 +0000 UTC m=+2106.678415403" Nov 03 22:42:30.192223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:30.191873 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:42:30.192223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:30.191934 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 22:42:30.262970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:30.262702 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:30.277459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:30.277155 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:30.372154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:30.372113 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:42:30.372452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:30.372419 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 03 22:42:30.557658 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:30.557619 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:30.558552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:30.558528 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:42:31.192168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.192140 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:42:31.192305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.192188 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 22:42:31.369704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.367913 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:31.369704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.369383 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:31.370323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.370307 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:42:31.396072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.396048 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:42:31.470621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.470588 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-envoy\") pod \"3da18026-d752-43f1-a018-99c54759fb56\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " Nov 03 22:42:31.470811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.470683 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-credential-socket\") pod \"3da18026-d752-43f1-a018-99c54759fb56\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " Nov 03 22:42:31.470811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.470728 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3da18026-d752-43f1-a018-99c54759fb56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da18026-d752-43f1-a018-99c54759fb56-istio-podinfo\") pod \"3da18026-d752-43f1-a018-99c54759fb56\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " Nov 03 22:42:31.470811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.470771 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-certs\") pod \"3da18026-d752-43f1-a018-99c54759fb56\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " Nov 03 22:42:31.471011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.470814 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-data\") pod \"3da18026-d752-43f1-a018-99c54759fb56\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " Nov 03 22:42:31.471011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.470840 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da18026-d752-43f1-a018-99c54759fb56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da18026-d752-43f1-a018-99c54759fb56-istiod-ca-cert\") pod \"3da18026-d752-43f1-a018-99c54759fb56\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " Nov 03 22:42:31.471011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.470960 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-istio-token\" (UniqueName: \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-istio-token\") pod \"3da18026-d752-43f1-a018-99c54759fb56\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " Nov 03 22:42:31.471167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471012 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-socket\") pod \"3da18026-d752-43f1-a018-99c54759fb56\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " Nov 03 22:42:31.471167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471025 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-credential-socket" pod "3da18026-d752-43f1-a018-99c54759fb56" (UID: "3da18026-d752-43f1-a018-99c54759fb56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:31.471167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471060 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-kube-api-access-kvk6x\" (UniqueName: \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-kube-api-access-kvk6x\") pod \"3da18026-d752-43f1-a018-99c54759fb56\" (UID: \"3da18026-d752-43f1-a018-99c54759fb56\") " Nov 03 22:42:31.471167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471128 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-certs" pod "3da18026-d752-43f1-a018-99c54759fb56" (UID: "3da18026-d752-43f1-a018-99c54759fb56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:31.471167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471143 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-data" pod "3da18026-d752-43f1-a018-99c54759fb56" (UID: "3da18026-d752-43f1-a018-99c54759fb56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:31.471435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471210 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da18026-d752-43f1-a018-99c54759fb56-istiod-ca-cert" pod "3da18026-d752-43f1-a018-99c54759fb56" (UID: "3da18026-d752-43f1-a018-99c54759fb56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:31.471435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471340 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-socket" pod "3da18026-d752-43f1-a018-99c54759fb56" (UID: "3da18026-d752-43f1-a018-99c54759fb56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:31.471787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471769 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:31.471787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471786 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:31.471948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471798 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da18026-d752-43f1-a018-99c54759fb56-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:31.471948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471814 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:31.471948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.471828 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:31.473326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.473305 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-kube-api-access-kvk6x" pod "3da18026-d752-43f1-a018-99c54759fb56" (UID: "3da18026-d752-43f1-a018-99c54759fb56"). InnerVolumeSpecName "kube-api-access-kvk6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:31.473530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.473507 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3da18026-d752-43f1-a018-99c54759fb56-istio-podinfo" pod "3da18026-d752-43f1-a018-99c54759fb56" (UID: "3da18026-d752-43f1-a018-99c54759fb56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:31.473626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.473531 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-envoy" pod "3da18026-d752-43f1-a018-99c54759fb56" (UID: "3da18026-d752-43f1-a018-99c54759fb56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:31.473626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.473537 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-istio-token" pod "3da18026-d752-43f1-a018-99c54759fb56" (UID: "3da18026-d752-43f1-a018-99c54759fb56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:31.566898 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.566861 4298 generic.go:356] "Generic (PLEG): container finished" podID="3da18026-d752-43f1-a018-99c54759fb56" containerID="2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1" exitCode=0 Nov 03 22:42:31.567107 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.566922 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" Nov 03 22:42:31.567107 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.566932 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" event={"ID":"3da18026-d752-43f1-a018-99c54759fb56","Type":"ContainerDied","Data":"2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1"} Nov 03 22:42:31.567107 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.566984 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w" event={"ID":"3da18026-d752-43f1-a018-99c54759fb56","Type":"ContainerDied","Data":"8ebe27575702e6e3e373f4a6a96c377d8b91bff524182b15cdc7b2887f930497"} Nov 03 22:42:31.567107 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.567005 4298 scope.go:122] "RemoveContainer" containerID="2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1" Nov 03 22:42:31.572926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.572900 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:31.573062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.572931 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvk6x\" (UniqueName: \"kubernetes.io/projected/3da18026-d752-43f1-a018-99c54759fb56-kube-api-access-kvk6x\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:31.573062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.572948 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da18026-d752-43f1-a018-99c54759fb56-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:31.573062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.572962 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da18026-d752-43f1-a018-99c54759fb56-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:31.579018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.578996 4298 scope.go:122] "RemoveContainer" containerID="58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb" Nov 03 22:42:31.588213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.588195 4298 scope.go:122] "RemoveContainer" containerID="2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1" Nov 03 22:42:31.588493 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:42:31.588472 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1\": container with ID starting with 2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1 not found: ID does not exist" containerID="2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1" Nov 03 22:42:31.588557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.588500 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1"} err="failed to get container status \"2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1\": rpc error: code = NotFound desc = could not find container \"2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1\": container with ID starting with 2c09fc3f914a486bcc248a719d075549dabb87ff6ee40ea97c9c3278d345c1a1 not found: ID does not exist" Nov 03 22:42:31.588557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.588520 4298 scope.go:122] "RemoveContainer" containerID="58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb" Nov 03 22:42:31.588797 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:42:31.588776 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb\": container with ID starting with 58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb not found: ID does not exist" containerID="58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb" Nov 03 22:42:31.588887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.588805 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb"} err="failed to get container status \"58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb\": rpc error: code = NotFound desc = could not find container \"58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb\": container with ID starting with 58ead6f738c8f190ff2da0ca18fa8d961fbae227d0c7c83df035d1df6b55a4fb not found: ID does not exist" Nov 03 22:42:31.863752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.863718 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w"] Nov 03 22:42:31.869106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:31.869077 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e7d36a-openshift-default-664ff4f4d7-zhw4w"] Nov 03 22:42:32.191838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:32.191756 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:42:32.191838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:32.191808 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 22:42:32.354668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:32.354464 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:32.684392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:32.684352 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:32.914074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:32.914042 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:33.155297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.155265 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da18026-d752-43f1-a018-99c54759fb56" path="/var/lib/kubelet/pods/3da18026-d752-43f1-a018-99c54759fb56/volumes" Nov 03 22:42:33.192628 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.192590 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:42:33.192827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.192672 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 22:42:33.353904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.353863 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n"] Nov 03 22:42:33.354119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.354093 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerName="istio-proxy" containerID="cri-o://adaf6d6f50909ba3777020012f453d72a1d8347d4988e9cedcb0d5946e1e769c" gracePeriod=30 Nov 03 22:42:33.359036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.359010 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn"] Nov 03 22:42:33.359433 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.359207 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerName="istio-proxy" containerID="cri-o://53a7b6639c0265b9a7fb6324b98269cf367f205d4948e3ab7ca3bae5658ce072" gracePeriod=30 Nov 03 22:42:33.363805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.363778 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn"] Nov 03 22:42:33.364014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.363980 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerName="istio-proxy" containerID="cri-o://d861d071e83a034c8d02b43f77684a502d311450c50ee064c117774d8ca1a751" gracePeriod=30 Nov 03 22:42:33.405911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.405828 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr"] Nov 03 22:42:33.406087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.406053 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerName="istio-proxy" containerID="cri-o://8b8846f20629bf353fc0e0215e624b2ba5daa00a0ce17a7ff6d1bc34bc997be1" gracePeriod=30 Nov 03 22:42:33.604295 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.604263 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:33.889450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:33.889403 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:34.201663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.198831 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:42:34.201663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.198894 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 22:42:34.246694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.245905 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l"] Nov 03 22:42:34.247671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.246953 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" Nov 03 22:42:34.247671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.246979 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" Nov 03 22:42:34.247671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.247219 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" Nov 03 22:42:34.247671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.247240 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" Nov 03 22:42:34.284728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.277517 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.317824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.316865 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l"] Nov 03 22:42:34.323055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.323024 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-credential-socket\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.333764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.333730 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-certs\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.334094 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.334074 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-envoy\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.334352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.334333 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istiod-ca-cert\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.334501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.334485 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-socket\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.334672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.334635 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-podinfo\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.334852 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.334834 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-token\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.334994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.334980 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-data\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.335281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.335262 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2bm9\" (UniqueName: \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-kube-api-access-l2bm9\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.340726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.340700 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c8f168-openshift-default-dockercfg-cfgbn\"" Nov 03 22:42:34.402688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.393123 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:34.476728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.464659 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-podinfo\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.476728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.464706 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-token\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.476728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.464748 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-data\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.476728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.464852 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2bm9\" (UniqueName: \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-kube-api-access-l2bm9\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.476728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.464891 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-credential-socket\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.476728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.464974 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-certs\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.476728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.465297 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-data\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.476728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.471352 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-credential-socket\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.476728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.471908 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-certs\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.482372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.481294 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-podinfo\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.482372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.465111 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-envoy\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.482372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.481814 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istiod-ca-cert\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.482372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.481876 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-socket\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.482372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.482321 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-socket\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.482801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.482528 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istiod-ca-cert\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.485766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.485470 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-envoy\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.556673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.543692 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2bm9\" (UniqueName: \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-kube-api-access-l2bm9\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.563184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.563154 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-token\") pod \"e2e-api-a-c8f168-openshift-default-786989b665-shc2l\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.653667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.649941 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:34.666676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.662760 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:34.888404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:34.888365 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:35.093456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.093366 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:35.267511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.267456 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l"] Nov 03 22:42:35.267701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.267613 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:35.268943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.268880 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:35.270050 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.270030 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:42:35.271047 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.271009 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:35.271166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.271144 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:35.271253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.271232 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:35.365721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.355202 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:35.552754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.552715 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx"] Nov 03 22:42:35.553770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.553745 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" Nov 03 22:42:35.553770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.553770 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da18026-d752-43f1-a018-99c54759fb56" containerName="istio-proxy" Nov 03 22:42:35.558339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.558316 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:42:35.608184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.608150 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx"] Nov 03 22:42:35.609597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.609568 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" event={"ID":"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2","Type":"ContainerStarted","Data":"87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413"} Nov 03 22:42:35.609767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.609631 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" event={"ID":"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2","Type":"ContainerStarted","Data":"8abb2b269616f21258b43e0bee94e39acd774c227db33801eb5e44a7dd156a19"} Nov 03 22:42:35.612511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.612474 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b30a14-dockercfg-6xnmx\"" Nov 03 22:42:35.614307 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.614284 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b30a14\"" Nov 03 22:42:35.651679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.651540 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:35.695675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.693035 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6639a62-b480-40e7-8c87-9aa9ba85160d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx\" (UID: \"c6639a62-b480-40e7-8c87-9aa9ba85160d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:42:35.695675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.693101 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5xbl\" (UniqueName: \"kubernetes.io/projected/c6639a62-b480-40e7-8c87-9aa9ba85160d-kube-api-access-q5xbl\") pod \"payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx\" (UID: \"c6639a62-b480-40e7-8c87-9aa9ba85160d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:42:35.798675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.796121 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6639a62-b480-40e7-8c87-9aa9ba85160d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx\" (UID: \"c6639a62-b480-40e7-8c87-9aa9ba85160d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:42:35.798675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.796183 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5xbl\" (UniqueName: \"kubernetes.io/projected/c6639a62-b480-40e7-8c87-9aa9ba85160d-kube-api-access-q5xbl\") pod \"payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx\" (UID: \"c6639a62-b480-40e7-8c87-9aa9ba85160d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:42:35.824682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.820395 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6639a62-b480-40e7-8c87-9aa9ba85160d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx\" (UID: \"c6639a62-b480-40e7-8c87-9aa9ba85160d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:42:35.870674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.869632 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5xbl\" (UniqueName: \"kubernetes.io/projected/c6639a62-b480-40e7-8c87-9aa9ba85160d-kube-api-access-q5xbl\") pod \"payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx\" (UID: \"c6639a62-b480-40e7-8c87-9aa9ba85160d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:42:35.888144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:35.885758 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:42:36.384627 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:42:36.384593 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6639a62_b480_40e7_8c87_9aa9ba85160d.slice/crio-d9de2762377dac5450a90e1e63f4e4d863c10582779ad066bbfc1088dc15a69f WatchSource:0}: Error finding container d9de2762377dac5450a90e1e63f4e4d863c10582779ad066bbfc1088dc15a69f: Status 404 returned error can't find the container with id d9de2762377dac5450a90e1e63f4e4d863c10582779ad066bbfc1088dc15a69f Nov 03 22:42:36.389164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.389120 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" podStartSLOduration=2.389104729 podStartE2EDuration="2.389104729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:35.856808518 +0000 UTC m=+2112.837542172" watchObservedRunningTime="2026-11-03 22:42:36.389104729 +0000 UTC m=+2113.369838381" Nov 03 22:42:36.390002 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.389979 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx"] Nov 03 22:42:36.617685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.617617 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" event={"ID":"c6639a62-b480-40e7-8c87-9aa9ba85160d","Type":"ContainerStarted","Data":"e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33"} Nov 03 22:42:36.618090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.617991 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:42:36.618090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.618035 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" event={"ID":"c6639a62-b480-40e7-8c87-9aa9ba85160d","Type":"ContainerStarted","Data":"d9de2762377dac5450a90e1e63f4e4d863c10582779ad066bbfc1088dc15a69f"} Nov 03 22:42:36.656973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.656894 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:36.719290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.719257 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9"] Nov 03 22:42:36.723901 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.723874 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:36.736453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.736408 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" podStartSLOduration=1.7363949920000001 podStartE2EDuration="1.736394992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:36.732482504 +0000 UTC m=+2113.713216158" watchObservedRunningTime="2026-11-03 22:42:36.736394992 +0000 UTC m=+2113.717128645" Nov 03 22:42:36.777061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.777026 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:42:36.777273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.777254 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:42:36.783533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.783502 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9"] Nov 03 22:42:36.820521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.820484 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:36.820521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.820532 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmp8r\" (UniqueName: \"kubernetes.io/projected/0d150f82-ce58-454e-b2a6-cd3abff079e4-kube-api-access-wmp8r\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:36.820857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.820832 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:36.922119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.922036 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:36.922119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.922074 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmp8r\" (UniqueName: \"kubernetes.io/projected/0d150f82-ce58-454e-b2a6-cd3abff079e4-kube-api-access-wmp8r\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:36.922374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.922206 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:36.922767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.922738 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:36.924825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.924802 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:36.982627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:36.982590 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmp8r\" (UniqueName: \"kubernetes.io/projected/0d150f82-ce58-454e-b2a6-cd3abff079e4-kube-api-access-wmp8r\") pod \"maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:37.038982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.038947 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:37.176134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.176057 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl"] Nov 03 22:42:37.182527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.182492 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.226369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.226340 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.226511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.226385 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.226511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.226450 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.226511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.226494 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.226696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.226677 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.226765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.226716 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.226897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.226880 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.226971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.226952 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.227088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.227074 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvggm\" (UniqueName: \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-kube-api-access-kvggm\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.271961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.271928 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl"] Nov 03 22:42:37.328955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.328921 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.328955 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.328956 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.329152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.329039 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.329152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.329080 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.329152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.329141 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvggm\" (UniqueName: \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-kube-api-access-kvggm\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.329265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.329206 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.329265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.329243 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.329359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.329273 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.329359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.329322 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.329446 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.329319 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.329613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.329581 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.329779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.329674 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.329779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.329754 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.330030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.330011 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.331271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.331252 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.331752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.331732 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.386112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.386083 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.386452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.386432 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvggm\" (UniqueName: \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-kube-api-access-kvggm\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.411376 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:42:37.411332 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d150f82_ce58_454e_b2a6_cd3abff079e4.slice/crio-225f8b336946c2279024678c4483b8f048956cf86ea03b6679c0b6dc74a8c45a WatchSource:0}: Error finding container 225f8b336946c2279024678c4483b8f048956cf86ea03b6679c0b6dc74a8c45a: Status 404 returned error can't find the container with id 225f8b336946c2279024678c4483b8f048956cf86ea03b6679c0b6dc74a8c45a Nov 03 22:42:37.416683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.416659 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9"] Nov 03 22:42:37.497988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.497963 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:37.628422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.628380 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" event={"ID":"0d150f82-ce58-454e-b2a6-cd3abff079e4","Type":"ContainerStarted","Data":"225f8b336946c2279024678c4483b8f048956cf86ea03b6679c0b6dc74a8c45a"} Nov 03 22:42:37.628887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.628750 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:37.629940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.629918 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:42:37.917980 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:42:37.917940 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84461fb7_32f0_48ea_b8dd_f7a03d6b3191.slice/crio-f1cd5641c7d99aa8184d0beb2e49f89bcdd824858589101e260fa7600d05830e WatchSource:0}: Error finding container f1cd5641c7d99aa8184d0beb2e49f89bcdd824858589101e260fa7600d05830e: Status 404 returned error can't find the container with id f1cd5641c7d99aa8184d0beb2e49f89bcdd824858589101e260fa7600d05830e Nov 03 22:42:37.921780 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.921737 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:37.921892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.921828 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:37.921892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.921869 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:37.924138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:37.924106 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl"] Nov 03 22:42:38.668790 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.668615 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" event={"ID":"84461fb7-32f0-48ea-b8dd-f7a03d6b3191","Type":"ContainerStarted","Data":"de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256"} Nov 03 22:42:38.668790 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.668713 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" event={"ID":"84461fb7-32f0-48ea-b8dd-f7a03d6b3191","Type":"ContainerStarted","Data":"f1cd5641c7d99aa8184d0beb2e49f89bcdd824858589101e260fa7600d05830e"} Nov 03 22:42:38.671819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.671792 4298 generic.go:356] "Generic (PLEG): container finished" podID="0d150f82-ce58-454e-b2a6-cd3abff079e4" containerID="e0469f3f93fc38be1fd4f2f591959db645dca9704547cd5c5917acb54ebf6108" exitCode=0 Nov 03 22:42:38.671940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.671878 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" event={"ID":"0d150f82-ce58-454e-b2a6-cd3abff079e4","Type":"ContainerDied","Data":"e0469f3f93fc38be1fd4f2f591959db645dca9704547cd5c5917acb54ebf6108"} Nov 03 22:42:38.673495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.673475 4298 generic.go:356] "Generic (PLEG): container finished" podID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerID="adaf6d6f50909ba3777020012f453d72a1d8347d4988e9cedcb0d5946e1e769c" exitCode=0 Nov 03 22:42:38.673593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.673578 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" event={"ID":"00d46a0c-7111-487a-862b-a89a8bf4ecce","Type":"ContainerDied","Data":"adaf6d6f50909ba3777020012f453d72a1d8347d4988e9cedcb0d5946e1e769c"} Nov 03 22:42:38.675237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.675217 4298 generic.go:356] "Generic (PLEG): container finished" podID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerID="d861d071e83a034c8d02b43f77684a502d311450c50ee064c117774d8ca1a751" exitCode=0 Nov 03 22:42:38.675315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.675297 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" event={"ID":"62d1ef14-ddb0-43bd-9315-3993276fd02f","Type":"ContainerDied","Data":"d861d071e83a034c8d02b43f77684a502d311450c50ee064c117774d8ca1a751"} Nov 03 22:42:38.677014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.676987 4298 generic.go:356] "Generic (PLEG): container finished" podID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerID="53a7b6639c0265b9a7fb6324b98269cf367f205d4948e3ab7ca3bae5658ce072" exitCode=0 Nov 03 22:42:38.677114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.677098 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" event={"ID":"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db","Type":"ContainerDied","Data":"53a7b6639c0265b9a7fb6324b98269cf367f205d4948e3ab7ca3bae5658ce072"} Nov 03 22:42:38.678689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.678666 4298 generic.go:356] "Generic (PLEG): container finished" podID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerID="8b8846f20629bf353fc0e0215e624b2ba5daa00a0ce17a7ff6d1bc34bc997be1" exitCode=0 Nov 03 22:42:38.678797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.678769 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" event={"ID":"2f0aca7b-d296-4cde-9146-d4a0d30c7c40","Type":"ContainerDied","Data":"8b8846f20629bf353fc0e0215e624b2ba5daa00a0ce17a7ff6d1bc34bc997be1"} Nov 03 22:42:38.724823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.724737 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:42:38.801022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.800965 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" podStartSLOduration=1.8009515569999999 podStartE2EDuration="1.800951557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:38.794527136 +0000 UTC m=+2115.775260781" watchObservedRunningTime="2026-11-03 22:42:38.800951557 +0000 UTC m=+2115.781685210" Nov 03 22:42:38.801247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.801228 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:42:38.870349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870325 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-certs\") pod \"00d46a0c-7111-487a-862b-a89a8bf4ecce\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " Nov 03 22:42:38.870467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870387 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-token\" (UniqueName: \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-token\") pod \"62d1ef14-ddb0-43bd-9315-3993276fd02f\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " Nov 03 22:42:38.870467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870432 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-data\") pod \"62d1ef14-ddb0-43bd-9315-3993276fd02f\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " Nov 03 22:42:38.870467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870452 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-certs\") pod \"62d1ef14-ddb0-43bd-9315-3993276fd02f\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " Nov 03 22:42:38.870562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870530 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-envoy\") pod \"62d1ef14-ddb0-43bd-9315-3993276fd02f\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " Nov 03 22:42:38.870562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870554 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-kube-api-access-fsqrv\" (UniqueName: \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-kube-api-access-fsqrv\") pod \"00d46a0c-7111-487a-862b-a89a8bf4ecce\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " Nov 03 22:42:38.870629 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870586 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-podinfo\") pod \"00d46a0c-7111-487a-862b-a89a8bf4ecce\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " Nov 03 22:42:38.870629 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870607 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-data\") pod \"00d46a0c-7111-487a-862b-a89a8bf4ecce\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " Nov 03 22:42:38.870745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870699 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d1ef14-ddb0-43bd-9315-3993276fd02f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d1ef14-ddb0-43bd-9315-3993276fd02f-istiod-ca-cert\") pod \"62d1ef14-ddb0-43bd-9315-3993276fd02f\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " Nov 03 22:42:38.870745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870728 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-socket\") pod \"62d1ef14-ddb0-43bd-9315-3993276fd02f\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " Nov 03 22:42:38.870835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870823 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-socket\") pod \"00d46a0c-7111-487a-862b-a89a8bf4ecce\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " Nov 03 22:42:38.870879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870846 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-credential-socket\") pod \"62d1ef14-ddb0-43bd-9315-3993276fd02f\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " Nov 03 22:42:38.870879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870864 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-kube-api-access-892rc\" (UniqueName: \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-kube-api-access-892rc\") pod \"62d1ef14-ddb0-43bd-9315-3993276fd02f\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " Nov 03 22:42:38.870951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870887 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-credential-socket\") pod \"00d46a0c-7111-487a-862b-a89a8bf4ecce\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " Nov 03 22:42:38.870951 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.870921 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-envoy\") pod \"00d46a0c-7111-487a-862b-a89a8bf4ecce\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " Nov 03 22:42:38.871310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.871294 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-podinfo\") pod \"62d1ef14-ddb0-43bd-9315-3993276fd02f\" (UID: \"62d1ef14-ddb0-43bd-9315-3993276fd02f\") " Nov 03 22:42:38.871359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.871335 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-token\" (UniqueName: \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-token\") pod \"00d46a0c-7111-487a-862b-a89a8bf4ecce\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " Nov 03 22:42:38.871391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.871365 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d46a0c-7111-487a-862b-a89a8bf4ecce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d46a0c-7111-487a-862b-a89a8bf4ecce-istiod-ca-cert\") pod \"00d46a0c-7111-487a-862b-a89a8bf4ecce\" (UID: \"00d46a0c-7111-487a-862b-a89a8bf4ecce\") " Nov 03 22:42:38.871999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.871963 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-certs" pod "00d46a0c-7111-487a-862b-a89a8bf4ecce" (UID: "00d46a0c-7111-487a-862b-a89a8bf4ecce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.872133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.872114 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.872441 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.872398 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-data" pod "00d46a0c-7111-487a-862b-a89a8bf4ecce" (UID: "00d46a0c-7111-487a-862b-a89a8bf4ecce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.872562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.872540 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d1ef14-ddb0-43bd-9315-3993276fd02f-istiod-ca-cert" pod "62d1ef14-ddb0-43bd-9315-3993276fd02f" (UID: "62d1ef14-ddb0-43bd-9315-3993276fd02f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:38.872634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.872613 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-certs" pod "62d1ef14-ddb0-43bd-9315-3993276fd02f" (UID: "62d1ef14-ddb0-43bd-9315-3993276fd02f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.872828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.872808 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-data" pod "62d1ef14-ddb0-43bd-9315-3993276fd02f" (UID: "62d1ef14-ddb0-43bd-9315-3993276fd02f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.872828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.872821 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-socket" pod "62d1ef14-ddb0-43bd-9315-3993276fd02f" (UID: "62d1ef14-ddb0-43bd-9315-3993276fd02f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.873008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.872991 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-socket" pod "00d46a0c-7111-487a-862b-a89a8bf4ecce" (UID: "00d46a0c-7111-487a-862b-a89a8bf4ecce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.873246 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.873157 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-credential-socket" pod "62d1ef14-ddb0-43bd-9315-3993276fd02f" (UID: "62d1ef14-ddb0-43bd-9315-3993276fd02f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.873394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.873371 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-podinfo" pod "00d46a0c-7111-487a-862b-a89a8bf4ecce" (UID: "00d46a0c-7111-487a-862b-a89a8bf4ecce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:38.873906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.873882 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-credential-socket" pod "00d46a0c-7111-487a-862b-a89a8bf4ecce" (UID: "00d46a0c-7111-487a-862b-a89a8bf4ecce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.874749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.874469 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d46a0c-7111-487a-862b-a89a8bf4ecce-istiod-ca-cert" pod "00d46a0c-7111-487a-862b-a89a8bf4ecce" (UID: "00d46a0c-7111-487a-862b-a89a8bf4ecce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:38.876122 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.876087 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-token" pod "62d1ef14-ddb0-43bd-9315-3993276fd02f" (UID: "62d1ef14-ddb0-43bd-9315-3993276fd02f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:38.876582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.876554 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-envoy" pod "00d46a0c-7111-487a-862b-a89a8bf4ecce" (UID: "00d46a0c-7111-487a-862b-a89a8bf4ecce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.876687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.876636 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-envoy" pod "62d1ef14-ddb0-43bd-9315-3993276fd02f" (UID: "62d1ef14-ddb0-43bd-9315-3993276fd02f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.877015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.876995 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-podinfo" pod "62d1ef14-ddb0-43bd-9315-3993276fd02f" (UID: "62d1ef14-ddb0-43bd-9315-3993276fd02f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:38.878051 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.878023 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-kube-api-access-fsqrv" pod "00d46a0c-7111-487a-862b-a89a8bf4ecce" (UID: "00d46a0c-7111-487a-862b-a89a8bf4ecce"). InnerVolumeSpecName "kube-api-access-fsqrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:38.878513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.878491 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-token" pod "00d46a0c-7111-487a-862b-a89a8bf4ecce" (UID: "00d46a0c-7111-487a-862b-a89a8bf4ecce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:38.880401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.880379 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-kube-api-access-892rc" pod "62d1ef14-ddb0-43bd-9315-3993276fd02f" (UID: "62d1ef14-ddb0-43bd-9315-3993276fd02f"). InnerVolumeSpecName "kube-api-access-892rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:38.885345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.885321 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:42:38.890183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.890160 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:42:38.973113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973083 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istiod-ca-cert\") pod \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " Nov 03 22:42:38.973314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973136 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-credential-socket\") pod \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " Nov 03 22:42:38.973314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973169 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-certs\") pod \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " Nov 03 22:42:38.973314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973209 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-podinfo\") pod \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " Nov 03 22:42:38.973314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973244 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-envoy\") pod \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " Nov 03 22:42:38.973314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973274 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-token\" (UniqueName: \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-token\") pod \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " Nov 03 22:42:38.973605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973314 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-socket\") pod \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " Nov 03 22:42:38.973605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973365 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-kube-api-access-rlsxs\" (UniqueName: \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-kube-api-access-rlsxs\") pod \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " Nov 03 22:42:38.973605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973437 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-socket\") pod \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " Nov 03 22:42:38.973605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973469 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-data\") pod \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " Nov 03 22:42:38.973605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973448 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-certs" pod "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" (UID: "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.973605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973512 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-credential-socket" pod "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" (UID: "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.973605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973529 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-podinfo\") pod \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " Nov 03 22:42:38.973605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973599 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-credential-socket\") pod \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " Nov 03 22:42:38.974055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973611 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-socket" pod "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" (UID: "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.974055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973668 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-envoy\") pod \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " Nov 03 22:42:38.974055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973719 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istiod-ca-cert\") pod \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " Nov 03 22:42:38.974055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973749 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-kube-api-access-hb6jn\" (UniqueName: \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-kube-api-access-hb6jn\") pod \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " Nov 03 22:42:38.974055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973809 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-certs\") pod \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " Nov 03 22:42:38.974055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973919 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-data\") pod \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\" (UID: \"2f0aca7b-d296-4cde-9146-d4a0d30c7c40\") " Nov 03 22:42:38.974372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974067 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-token\" (UniqueName: \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-token\") pod \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\" (UID: \"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db\") " Nov 03 22:42:38.974372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973542 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istiod-ca-cert" pod "2f0aca7b-d296-4cde-9146-d4a0d30c7c40" (UID: "2f0aca7b-d296-4cde-9146-d4a0d30c7c40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:38.974372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973802 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-data" pod "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" (UID: "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.974372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.973849 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-socket" pod "2f0aca7b-d296-4cde-9146-d4a0d30c7c40" (UID: "2f0aca7b-d296-4cde-9146-d4a0d30c7c40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.974372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974142 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-certs" pod "2f0aca7b-d296-4cde-9146-d4a0d30c7c40" (UID: "2f0aca7b-d296-4cde-9146-d4a0d30c7c40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.974372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974163 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istiod-ca-cert" pod "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" (UID: "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:38.974372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974337 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-credential-socket" pod "2f0aca7b-d296-4cde-9146-d4a0d30c7c40" (UID: "2f0aca7b-d296-4cde-9146-d4a0d30c7c40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.974734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974491 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-data" pod "2f0aca7b-d296-4cde-9146-d4a0d30c7c40" (UID: "2f0aca7b-d296-4cde-9146-d4a0d30c7c40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974825 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974862 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974878 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsqrv\" (UniqueName: \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-kube-api-access-fsqrv\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974895 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974908 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974920 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974933 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d1ef14-ddb0-43bd-9315-3993276fd02f-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974946 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974960 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974975 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.974989 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975003 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975017 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975031 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-892rc\" (UniqueName: \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-kube-api-access-892rc\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975044 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975059 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975072 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975086 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975100 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d46a0c-7111-487a-862b-a89a8bf4ecce-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975114 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975127 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d46a0c-7111-487a-862b-a89a8bf4ecce-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975140 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975155 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975169 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975183 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975196 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d1ef14-ddb0-43bd-9315-3993276fd02f-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.975245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.975212 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.976858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.976009 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-podinfo" pod "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" (UID: "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:38.976858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.976370 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-token" pod "2f0aca7b-d296-4cde-9146-d4a0d30c7c40" (UID: "2f0aca7b-d296-4cde-9146-d4a0d30c7c40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:38.976858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.976515 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-kube-api-access-rlsxs" pod "2f0aca7b-d296-4cde-9146-d4a0d30c7c40" (UID: "2f0aca7b-d296-4cde-9146-d4a0d30c7c40"). InnerVolumeSpecName "kube-api-access-rlsxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:38.976858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.976779 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-podinfo" pod "2f0aca7b-d296-4cde-9146-d4a0d30c7c40" (UID: "2f0aca7b-d296-4cde-9146-d4a0d30c7c40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:38.976858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.976853 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-envoy" pod "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" (UID: "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.977067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.976877 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-token" pod "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" (UID: "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:38.977067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.977035 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-envoy" pod "2f0aca7b-d296-4cde-9146-d4a0d30c7c40" (UID: "2f0aca7b-d296-4cde-9146-d4a0d30c7c40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:38.977672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:38.977637 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-kube-api-access-hb6jn" pod "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" (UID: "73e8ca4d-4ac4-4248-be72-7a5c4c12e8db"). InnerVolumeSpecName "kube-api-access-hb6jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:39.076893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.076848 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:39.076893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.076887 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:39.076893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.076902 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:39.077219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.076916 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:39.077219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.076933 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlsxs\" (UniqueName: \"kubernetes.io/projected/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-kube-api-access-rlsxs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:39.077219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.076949 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:39.077219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.076963 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0aca7b-d296-4cde-9146-d4a0d30c7c40-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:39.077219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.076976 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb6jn\" (UniqueName: \"kubernetes.io/projected/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db-kube-api-access-hb6jn\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:39.498725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.498687 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:39.499899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.499871 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:42:39.500019 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.499919 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" podUID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 22:42:39.688037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.688006 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" Nov 03 22:42:39.688554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.688042 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr" event={"ID":"2f0aca7b-d296-4cde-9146-d4a0d30c7c40","Type":"ContainerDied","Data":"6dfb52409d9448e2c23a01052816d76a43f63d18d9a5efbdc4c33f1cb2251c37"} Nov 03 22:42:39.688554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.688110 4298 scope.go:122] "RemoveContainer" containerID="8b8846f20629bf353fc0e0215e624b2ba5daa00a0ce17a7ff6d1bc34bc997be1" Nov 03 22:42:39.689719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.689700 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" Nov 03 22:42:39.689817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.689762 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n" event={"ID":"00d46a0c-7111-487a-862b-a89a8bf4ecce","Type":"ContainerDied","Data":"fb4f9eff3fed00f5af7fa6168d84ae93f4e64d280730fec3a6ef392fb0f30236"} Nov 03 22:42:39.691758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.691726 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" Nov 03 22:42:39.691758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.691750 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn" event={"ID":"62d1ef14-ddb0-43bd-9315-3993276fd02f","Type":"ContainerDied","Data":"72342a20d5121f8dcf01032e649819c95fde0ed6159d96b3640310f7f05cd4d3"} Nov 03 22:42:39.693395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.693380 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" Nov 03 22:42:39.693475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.693409 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn" event={"ID":"73e8ca4d-4ac4-4248-be72-7a5c4c12e8db","Type":"ContainerDied","Data":"b0a464a33b9108e30de1acddd1d5b3c0f46db159f042b3eb2f206553d6a2e505"} Nov 03 22:42:39.702474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.702455 4298 scope.go:122] "RemoveContainer" containerID="adaf6d6f50909ba3777020012f453d72a1d8347d4988e9cedcb0d5946e1e769c" Nov 03 22:42:39.712204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.712182 4298 scope.go:122] "RemoveContainer" containerID="d861d071e83a034c8d02b43f77684a502d311450c50ee064c117774d8ca1a751" Nov 03 22:42:39.721986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.721964 4298 scope.go:122] "RemoveContainer" containerID="53a7b6639c0265b9a7fb6324b98269cf367f205d4948e3ab7ca3bae5658ce072" Nov 03 22:42:39.760246 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.760219 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:39.882301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.882265 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn"] Nov 03 22:42:39.885157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.885134 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-td7bn"] Nov 03 22:42:39.913613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.913588 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:39.954364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.954328 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:39.989291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.989261 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr"] Nov 03 22:42:39.991619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.991592 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-xks-ca\") pod \"0d150f82-ce58-454e-b2a6-cd3abff079e4\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " Nov 03 22:42:39.991779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.991743 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d150f82-ce58-454e-b2a6-cd3abff079e4-kube-api-access-wmp8r\" (UniqueName: \"kubernetes.io/projected/0d150f82-ce58-454e-b2a6-cd3abff079e4-kube-api-access-wmp8r\") pod \"0d150f82-ce58-454e-b2a6-cd3abff079e4\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " Nov 03 22:42:39.991835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.991810 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-openshift-ca\") pod \"0d150f82-ce58-454e-b2a6-cd3abff079e4\" (UID: \"0d150f82-ce58-454e-b2a6-cd3abff079e4\") " Nov 03 22:42:39.992335 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.992305 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-openshift-ca" pod "0d150f82-ce58-454e-b2a6-cd3abff079e4" (UID: "0d150f82-ce58-454e-b2a6-cd3abff079e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:39.993800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.993776 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-xks-ca" pod "0d150f82-ce58-454e-b2a6-cd3abff079e4" (UID: "0d150f82-ce58-454e-b2a6-cd3abff079e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:39.993875 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.993812 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d150f82-ce58-454e-b2a6-cd3abff079e4-kube-api-access-wmp8r" pod "0d150f82-ce58-454e-b2a6-cd3abff079e4" (UID: "0d150f82-ce58-454e-b2a6-cd3abff079e4"). InnerVolumeSpecName "kube-api-access-wmp8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:39.995066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:39.995049 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gkrnr"] Nov 03 22:42:40.094597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.093999 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:40.094597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.094055 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmp8r\" (UniqueName: \"kubernetes.io/projected/0d150f82-ce58-454e-b2a6-cd3abff079e4-kube-api-access-wmp8r\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:40.094597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.094080 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d150f82-ce58-454e-b2a6-cd3abff079e4-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:40.123564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.123529 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n"] Nov 03 22:42:40.159674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.159621 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:40.165028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.164995 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6tj7n"] Nov 03 22:42:40.256830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.256789 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn"] Nov 03 22:42:40.261458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.261432 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-mfcpn"] Nov 03 22:42:40.409295 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.409187 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:40.498318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.498283 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:42:40.498474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.498341 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" podUID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 22:42:40.597216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.597184 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:40.704521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.704495 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" Nov 03 22:42:40.704521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.704505 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9" event={"ID":"0d150f82-ce58-454e-b2a6-cd3abff079e4","Type":"ContainerDied","Data":"225f8b336946c2279024678c4483b8f048956cf86ea03b6679c0b6dc74a8c45a"} Nov 03 22:42:40.705003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.704552 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="225f8b336946c2279024678c4483b8f048956cf86ea03b6679c0b6dc74a8c45a" Nov 03 22:42:40.735758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.735728 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v"] Nov 03 22:42:40.736940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.736914 4298 cpu_manager.go:414] "containerMap: removing container" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerName="istio-proxy" Nov 03 22:42:40.737114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.737099 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerName="istio-proxy" Nov 03 22:42:40.737233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.737220 4298 cpu_manager.go:414] "containerMap: removing container" podUID="0d150f82-ce58-454e-b2a6-cd3abff079e4" containerName="revoke-keys" Nov 03 22:42:40.737339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.737328 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d150f82-ce58-454e-b2a6-cd3abff079e4" containerName="revoke-keys" Nov 03 22:42:40.737451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.737440 4298 cpu_manager.go:414] "containerMap: removing container" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerName="istio-proxy" Nov 03 22:42:40.737530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.737518 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerName="istio-proxy" Nov 03 22:42:40.737625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.737614 4298 cpu_manager.go:414] "containerMap: removing container" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerName="istio-proxy" Nov 03 22:42:40.737741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.737729 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerName="istio-proxy" Nov 03 22:42:40.737829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.737819 4298 cpu_manager.go:414] "containerMap: removing container" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerName="istio-proxy" Nov 03 22:42:40.737903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.737892 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerName="istio-proxy" Nov 03 22:42:40.738169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.738154 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d150f82-ce58-454e-b2a6-cd3abff079e4" containerName="revoke-keys" Nov 03 22:42:40.738272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.738261 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" containerName="istio-proxy" Nov 03 22:42:40.738361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.738351 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" containerName="istio-proxy" Nov 03 22:42:40.738441 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.738432 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" containerName="istio-proxy" Nov 03 22:42:40.738527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.738518 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" containerName="istio-proxy" Nov 03 22:42:40.756366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.756311 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:42:40.804667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.803988 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v"] Nov 03 22:42:40.810469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.810438 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1b2fb1\"" Nov 03 22:42:40.810687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.810572 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1b2fb1-dockercfg-cf6gq\"" Nov 03 22:42:40.857668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.819518 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt72w\" (UniqueName: \"kubernetes.io/projected/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-kube-api-access-pt72w\") pod \"payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v\" (UID: \"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:42:40.857668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.819561 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v\" (UID: \"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:42:40.930669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.923614 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt72w\" (UniqueName: \"kubernetes.io/projected/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-kube-api-access-pt72w\") pod \"payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v\" (UID: \"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:42:40.930669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.923692 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v\" (UID: \"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:42:40.930669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.924896 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v\" (UID: \"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:42:40.960717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.960486 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb"] Nov 03 22:42:40.982975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.981729 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:42:40.987293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:40.987269 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt72w\" (UniqueName: \"kubernetes.io/projected/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-kube-api-access-pt72w\") pod \"payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v\" (UID: \"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:42:41.002062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.002036 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb"] Nov 03 22:42:41.081698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.057290 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3bfd04-fff1-46d3-865e-b559ac337802-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb\" (UID: \"df3bfd04-fff1-46d3-865e-b559ac337802\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:42:41.081698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.057532 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d52sf\" (UniqueName: \"kubernetes.io/projected/df3bfd04-fff1-46d3-865e-b559ac337802-kube-api-access-d52sf\") pod \"payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb\" (UID: \"df3bfd04-fff1-46d3-865e-b559ac337802\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:42:41.081698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.078796 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:42:41.158691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.155082 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d46a0c-7111-487a-862b-a89a8bf4ecce" path="/var/lib/kubelet/pods/00d46a0c-7111-487a-862b-a89a8bf4ecce/volumes" Nov 03 22:42:41.158691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.155916 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0aca7b-d296-4cde-9146-d4a0d30c7c40" path="/var/lib/kubelet/pods/2f0aca7b-d296-4cde-9146-d4a0d30c7c40/volumes" Nov 03 22:42:41.158691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.156628 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d1ef14-ddb0-43bd-9315-3993276fd02f" path="/var/lib/kubelet/pods/62d1ef14-ddb0-43bd-9315-3993276fd02f/volumes" Nov 03 22:42:41.158691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.157589 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e8ca4d-4ac4-4248-be72-7a5c4c12e8db" path="/var/lib/kubelet/pods/73e8ca4d-4ac4-4248-be72-7a5c4c12e8db/volumes" Nov 03 22:42:41.159389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.159189 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3bfd04-fff1-46d3-865e-b559ac337802-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb\" (UID: \"df3bfd04-fff1-46d3-865e-b559ac337802\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:42:41.159389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.159314 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d52sf\" (UniqueName: \"kubernetes.io/projected/df3bfd04-fff1-46d3-865e-b559ac337802-kube-api-access-d52sf\") pod \"payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb\" (UID: \"df3bfd04-fff1-46d3-865e-b559ac337802\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:42:41.160773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.160456 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3bfd04-fff1-46d3-865e-b559ac337802-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb\" (UID: \"df3bfd04-fff1-46d3-865e-b559ac337802\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:42:41.217803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.217729 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d52sf\" (UniqueName: \"kubernetes.io/projected/df3bfd04-fff1-46d3-865e-b559ac337802-kube-api-access-d52sf\") pod \"payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb\" (UID: \"df3bfd04-fff1-46d3-865e-b559ac337802\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:42:41.311616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.311582 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:42:41.498857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.498711 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:42:41.498857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.498824 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" podUID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 22:42:41.509529 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:42:41.509501 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1181ab5_ba26_4ec7_94cb_f9cd9d1eef07.slice/crio-b868cc72fc12bb7d0e577fcc62b7952827e2c72923f8bed4a7026001092fcc10 WatchSource:0}: Error finding container b868cc72fc12bb7d0e577fcc62b7952827e2c72923f8bed4a7026001092fcc10: Status 404 returned error can't find the container with id b868cc72fc12bb7d0e577fcc62b7952827e2c72923f8bed4a7026001092fcc10 Nov 03 22:42:41.514300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.514277 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v"] Nov 03 22:42:41.707858 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:42:41.707821 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3bfd04_fff1_46d3_865e_b559ac337802.slice/crio-3b836bd716c2613293bc44d90eb1db23e93486e8623057536b9823d3c2020cf2 WatchSource:0}: Error finding container 3b836bd716c2613293bc44d90eb1db23e93486e8623057536b9823d3c2020cf2: Status 404 returned error can't find the container with id 3b836bd716c2613293bc44d90eb1db23e93486e8623057536b9823d3c2020cf2 Nov 03 22:42:41.714159 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.714133 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb"] Nov 03 22:42:41.714909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.714880 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" event={"ID":"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07","Type":"ContainerStarted","Data":"be61ee6775ca50352d51ff875a16f120090849d7d339038abe15a0e41232f2bc"} Nov 03 22:42:41.715015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.714928 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" event={"ID":"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07","Type":"ContainerStarted","Data":"b868cc72fc12bb7d0e577fcc62b7952827e2c72923f8bed4a7026001092fcc10"} Nov 03 22:42:41.715015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.714964 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:42:41.856790 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:41.856717 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" podStartSLOduration=1.8566977329999999 podStartE2EDuration="1.856697733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:41.839114196 +0000 UTC m=+2118.819847862" watchObservedRunningTime="2026-11-03 22:42:41.856697733 +0000 UTC m=+2118.837431387" Nov 03 22:42:42.498253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:42.498223 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:42:42.498515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:42.498271 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" podUID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 22:42:42.723460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:42.723424 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" event={"ID":"df3bfd04-fff1-46d3-865e-b559ac337802","Type":"ContainerStarted","Data":"4ce5366c9ec168d5736c5076a60962a101d8a6c627ec8308a0bbee4e16e36b6d"} Nov 03 22:42:42.724003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:42.723479 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" event={"ID":"df3bfd04-fff1-46d3-865e-b559ac337802","Type":"ContainerStarted","Data":"3b836bd716c2613293bc44d90eb1db23e93486e8623057536b9823d3c2020cf2"} Nov 03 22:42:42.724003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:42.723706 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:42:43.498882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:43.498851 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:42:43.499075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:43.498915 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" podUID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 22:42:44.560961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:44.560932 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:44.562411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:44.562390 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:44.563862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:44.563839 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:42:44.690921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:44.690885 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:44.741123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:44.741062 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" podStartSLOduration=4.741043162 podStartE2EDuration="4.741043162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:42.849333166 +0000 UTC m=+2119.830066816" watchObservedRunningTime="2026-11-03 22:42:44.741043162 +0000 UTC m=+2121.721776816" Nov 03 22:42:46.451600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:46.451565 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:46.754308 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:46.754184 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:46.933676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:46.933631 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv"] Nov 03 22:42:46.938157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:46.938133 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a92c9e-kkddv"] Nov 03 22:42:46.948103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:46.948072 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:47.138124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:47.138041 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab38f8d8-386d-4b26-958c-d51d4382180b" path="/var/lib/kubelet/pods/ab38f8d8-386d-4b26-958c-d51d4382180b/volumes" Nov 03 22:42:47.386408 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:47.386383 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:47.553605 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:42:47.553570 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:42:47.632157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:47.632130 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:42:51.867903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:51.867860 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:52.060491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.060459 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:52.284635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.275868 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:52.313538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.313508 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m"] Nov 03 22:42:52.318201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.318181 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:42:52.437329 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.435161 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c8f168\"" Nov 03 22:42:52.437329 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.435237 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c8f168-dockercfg-vdlqx\"" Nov 03 22:42:52.438708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.438667 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78598387-4616-4d8f-9d4b-acb3f5331af6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m\" (UID: \"78598387-4616-4d8f-9d4b-acb3f5331af6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:42:52.439091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.439066 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfk7t\" (UniqueName: \"kubernetes.io/projected/78598387-4616-4d8f-9d4b-acb3f5331af6-kube-api-access-kfk7t\") pod \"payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m\" (UID: \"78598387-4616-4d8f-9d4b-acb3f5331af6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:42:52.447739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.447713 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m"] Nov 03 22:42:52.498759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.498712 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg"] Nov 03 22:42:52.509070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.508958 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:42:52.527868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.527834 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg"] Nov 03 22:42:52.549614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.547986 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78598387-4616-4d8f-9d4b-acb3f5331af6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m\" (UID: \"78598387-4616-4d8f-9d4b-acb3f5331af6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:42:52.549614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.548261 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfk7t\" (UniqueName: \"kubernetes.io/projected/78598387-4616-4d8f-9d4b-acb3f5331af6-kube-api-access-kfk7t\") pod \"payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m\" (UID: \"78598387-4616-4d8f-9d4b-acb3f5331af6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:42:52.549614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.549553 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78598387-4616-4d8f-9d4b-acb3f5331af6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m\" (UID: \"78598387-4616-4d8f-9d4b-acb3f5331af6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:42:52.607371 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.607335 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfk7t\" (UniqueName: \"kubernetes.io/projected/78598387-4616-4d8f-9d4b-acb3f5331af6-kube-api-access-kfk7t\") pod \"payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m\" (UID: \"78598387-4616-4d8f-9d4b-acb3f5331af6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:42:52.633340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.633304 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:42:52.649953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.649919 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg\" (UID: \"e0ac7245-3eba-4cfe-bbff-58b56adf7b48\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:42:52.650080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.650058 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svw8z\" (UniqueName: \"kubernetes.io/projected/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-kube-api-access-svw8z\") pod \"payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg\" (UID: \"e0ac7245-3eba-4cfe-bbff-58b56adf7b48\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:42:52.745965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.745896 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:42:52.751561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.751534 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg\" (UID: \"e0ac7245-3eba-4cfe-bbff-58b56adf7b48\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:42:52.751744 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.751597 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svw8z\" (UniqueName: \"kubernetes.io/projected/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-kube-api-access-svw8z\") pod \"payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg\" (UID: \"e0ac7245-3eba-4cfe-bbff-58b56adf7b48\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:42:52.752536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.752503 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg\" (UID: \"e0ac7245-3eba-4cfe-bbff-58b56adf7b48\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:42:52.811001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.810926 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svw8z\" (UniqueName: \"kubernetes.io/projected/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-kube-api-access-svw8z\") pod \"payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg\" (UID: \"e0ac7245-3eba-4cfe-bbff-58b56adf7b48\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:42:52.855341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:52.855306 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:42:53.017579 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:42:53.017547 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78598387_4616_4d8f_9d4b_acb3f5331af6.slice/crio-ef3e2f8ecea6372d0b0bc0ce88b7cb8741cb4c72ed3a6cd25c89a87449414e4a WatchSource:0}: Error finding container ef3e2f8ecea6372d0b0bc0ce88b7cb8741cb4c72ed3a6cd25c89a87449414e4a: Status 404 returned error can't find the container with id ef3e2f8ecea6372d0b0bc0ce88b7cb8741cb4c72ed3a6cd25c89a87449414e4a Nov 03 22:42:53.023691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.023664 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m"] Nov 03 22:42:53.241266 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:42:53.241229 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ac7245_3eba_4cfe_bbff_58b56adf7b48.slice/crio-a55f31a663761bb83c959786ee74efa9d8ac0c6337f2e0a251bf8d60b37f8c5c WatchSource:0}: Error finding container a55f31a663761bb83c959786ee74efa9d8ac0c6337f2e0a251bf8d60b37f8c5c: Status 404 returned error can't find the container with id a55f31a663761bb83c959786ee74efa9d8ac0c6337f2e0a251bf8d60b37f8c5c Nov 03 22:42:53.250814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.246998 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg"] Nov 03 22:42:53.487599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.487564 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:53.677287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.677206 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:53.757765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.757277 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:42:53.831698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.831634 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" event={"ID":"78598387-4616-4d8f-9d4b-acb3f5331af6","Type":"ContainerStarted","Data":"e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d"} Nov 03 22:42:53.831900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.831728 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" event={"ID":"78598387-4616-4d8f-9d4b-acb3f5331af6","Type":"ContainerStarted","Data":"ef3e2f8ecea6372d0b0bc0ce88b7cb8741cb4c72ed3a6cd25c89a87449414e4a"} Nov 03 22:42:53.831900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.831758 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:42:53.833527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.833501 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" event={"ID":"e0ac7245-3eba-4cfe-bbff-58b56adf7b48","Type":"ContainerStarted","Data":"7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877"} Nov 03 22:42:53.833680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.833541 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" event={"ID":"e0ac7245-3eba-4cfe-bbff-58b56adf7b48","Type":"ContainerStarted","Data":"a55f31a663761bb83c959786ee74efa9d8ac0c6337f2e0a251bf8d60b37f8c5c"} Nov 03 22:42:53.833680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.833674 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:42:53.988324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.988288 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:53.996522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:53.996474 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" podStartSLOduration=1.996462516 podStartE2EDuration="1.996462516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:53.995520844 +0000 UTC m=+2130.976254496" watchObservedRunningTime="2026-11-03 22:42:53.996462516 +0000 UTC m=+2130.977196168" Nov 03 22:42:54.125920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:54.125883 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk"] Nov 03 22:42:54.126381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:54.126149 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" containerID="cri-o://d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4" gracePeriod=30 Nov 03 22:42:54.143607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:54.143562 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" podStartSLOduration=2.143548954 podStartE2EDuration="2.143548954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:54.119976005 +0000 UTC m=+2131.100709659" watchObservedRunningTime="2026-11-03 22:42:54.143548954 +0000 UTC m=+2131.124282645" Nov 03 22:42:54.184798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:54.184763 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:54.392528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:54.392435 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:55.082918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.080217 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:55.595385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.595355 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb"] Nov 03 22:42:55.599909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.599888 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.661740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.661000 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-97965e-openshift-default-dockercfg-9zj4c\"" Nov 03 22:42:55.699418 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.699377 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-podinfo\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.808546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.806972 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-certs\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.808546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.807023 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c3301f-7e55-4e81-b5ce-420873ef1242-istiod-ca-cert\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.808546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.807064 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-token\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.808546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.807148 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-data\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.808546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.807208 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-socket\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.808546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.807248 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-envoy\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.808546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.807305 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-credential-socket\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.808546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.807376 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsbpd\" (UniqueName: \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-kube-api-access-vsbpd\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.808546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.746673 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:55.808546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.805737 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb"] Nov 03 22:42:55.909313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.909212 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-podinfo\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.909499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.909316 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-certs\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.909499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.909348 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c3301f-7e55-4e81-b5ce-420873ef1242-istiod-ca-cert\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.909499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.909384 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-token\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.909499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.909453 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-data\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.909766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.909505 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-socket\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.909766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.909543 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-envoy\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.909766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.909595 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-credential-socket\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.909766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.909676 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsbpd\" (UniqueName: \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-kube-api-access-vsbpd\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.910003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.909846 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-certs\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.910079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.910053 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-socket\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.910152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.910129 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-credential-socket\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.910200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.910153 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-data\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.910265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.910244 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c3301f-7e55-4e81-b5ce-420873ef1242-istiod-ca-cert\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.912458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.912434 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-envoy\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.912676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.912633 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-podinfo\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.968621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.968586 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsbpd\" (UniqueName: \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-kube-api-access-vsbpd\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:55.969032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:55.969009 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-token\") pod \"e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:56.229547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:56.229515 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:56.383051 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:56.383015 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:56.624530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:56.624439 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:56.667179 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:42:56.667144 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c3301f_7e55_4e81_b5ce_420873ef1242.slice/crio-90c0050bad3a647595f6a099be3e00fb073742650e3f859a167fb6fc6eced462 WatchSource:0}: Error finding container 90c0050bad3a647595f6a099be3e00fb073742650e3f859a167fb6fc6eced462: Status 404 returned error can't find the container with id 90c0050bad3a647595f6a099be3e00fb073742650e3f859a167fb6fc6eced462 Nov 03 22:42:56.670310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:56.670266 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:56.670413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:56.670360 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:56.670413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:56.670404 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:42:56.671376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:56.671357 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb"] Nov 03 22:42:56.880335 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:56.880234 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" event={"ID":"e6c3301f-7e55-4e81-b5ce-420873ef1242","Type":"ContainerStarted","Data":"5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726"} Nov 03 22:42:56.880612 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:56.880590 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" event={"ID":"e6c3301f-7e55-4e81-b5ce-420873ef1242","Type":"ContainerStarted","Data":"90c0050bad3a647595f6a099be3e00fb073742650e3f859a167fb6fc6eced462"} Nov 03 22:42:57.009657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:57.009576 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" podStartSLOduration=2.009556948 podStartE2EDuration="2.009556948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:57.002514298 +0000 UTC m=+2133.983247951" watchObservedRunningTime="2026-11-03 22:42:57.009556948 +0000 UTC m=+2133.990290605" Nov 03 22:42:57.232859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:57.232814 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:57.236921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:57.236883 4298 patch_prober.go:29] interesting pod/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 03 22:42:57.237074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:57.236943 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 03 22:42:58.234878 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:58.234847 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:58.241960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:58.241924 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:58.480813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:58.480777 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:58.674612 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:58.674497 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:58.811364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:58.811332 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s"] Nov 03 22:42:58.816230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:58.816209 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca6853-67v6s"] Nov 03 22:42:58.867836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:58.867805 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:58.895494 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:58.895471 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:58.896299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:58.896283 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:42:59.138377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.138341 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b0b26c-488c-4e62-8f5f-603d0e3d7e98" path="/var/lib/kubelet/pods/b6b0b26c-488c-4e62-8f5f-603d0e3d7e98/volumes" Nov 03 22:42:59.149459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.149432 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:59.410564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.410500 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:42:59.520592 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.520561 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-data\") pod \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " Nov 03 22:42:59.520794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.520600 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-credential-socket\") pod \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " Nov 03 22:42:59.520794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.520691 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-kube-api-access-qdsb4\" (UniqueName: \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-kube-api-access-qdsb4\") pod \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " Nov 03 22:42:59.520794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.520748 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-podinfo\") pod \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " Nov 03 22:42:59.520966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.520868 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-socket\") pod \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " Nov 03 22:42:59.520966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.520921 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-token\" (UniqueName: \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-token\") pod \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " Nov 03 22:42:59.520966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.520875 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-data" pod "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" (UID: "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:59.521125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.520977 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-credential-socket" pod "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" (UID: "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:59.521125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.521002 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istiod-ca-cert\") pod \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " Nov 03 22:42:59.521125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.521082 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-envoy\") pod \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " Nov 03 22:42:59.521272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.521130 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-socket" pod "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" (UID: "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:59.521272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.521191 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-certs\") pod \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\" (UID: \"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0\") " Nov 03 22:42:59.521387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.521369 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istiod-ca-cert" pod "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" (UID: "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:59.521527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.521503 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-certs" pod "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" (UID: "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:59.522118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.522090 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.522237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.522120 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.522237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.522137 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.522237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.522151 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.522237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.522165 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.523109 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.523088 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-podinfo" pod "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" (UID: "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:59.523212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.523127 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-kube-api-access-qdsb4" pod "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" (UID: "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0"). InnerVolumeSpecName "kube-api-access-qdsb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:59.523212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.523153 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-token" pod "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" (UID: "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:59.523304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.523293 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-envoy" pod "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" (UID: "a1c9d454-dbe9-4e6b-ab20-71b7388cedb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:59.624027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.623993 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdsb4\" (UniqueName: \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-kube-api-access-qdsb4\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.624027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.624024 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.624027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.624034 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.624257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.624044 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.903342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.903304 4298 generic.go:356] "Generic (PLEG): container finished" podID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerID="d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4" exitCode=0 Nov 03 22:42:59.903549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.903353 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" Nov 03 22:42:59.903549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.903389 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" event={"ID":"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0","Type":"ContainerDied","Data":"d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4"} Nov 03 22:42:59.903549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.903459 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk" event={"ID":"a1c9d454-dbe9-4e6b-ab20-71b7388cedb0","Type":"ContainerDied","Data":"a8e806485bf45dd4a9ebbcea06fab24b020bc0060700aba18ef67328c50ed598"} Nov 03 22:42:59.903549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.903488 4298 scope.go:122] "RemoveContainer" containerID="d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4" Nov 03 22:42:59.925991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.925962 4298 scope.go:122] "RemoveContainer" containerID="f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee" Nov 03 22:42:59.936205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.936178 4298 scope.go:122] "RemoveContainer" containerID="d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4" Nov 03 22:42:59.936536 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:42:59.936503 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4\": container with ID starting with d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4 not found: ID does not exist" containerID="d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4" Nov 03 22:42:59.936607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.936548 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4"} err="failed to get container status \"d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4\": rpc error: code = NotFound desc = could not find container \"d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4\": container with ID starting with d6156d0f26110cc2f53deb4939abd9ce2dd6d51d7aba5325ab8f22ac1af859c4 not found: ID does not exist" Nov 03 22:42:59.936607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.936573 4298 scope.go:122] "RemoveContainer" containerID="f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee" Nov 03 22:42:59.936871 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:42:59.936852 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee\": container with ID starting with f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee not found: ID does not exist" containerID="f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee" Nov 03 22:42:59.936953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:42:59.936877 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee"} err="failed to get container status \"f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee\": rpc error: code = NotFound desc = could not find container \"f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee\": container with ID starting with f089b7ae2fe61a2a5ff0bc4e525262fef48fe451d113ace2571bec42633833ee not found: ID does not exist" Nov 03 22:43:00.073541 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:00.073505 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk"] Nov 03 22:43:00.123630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:00.123601 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a92c9e-openshift-default-68978fbbc5-8nkzk"] Nov 03 22:43:01.137639 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:01.137609 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" path="/var/lib/kubelet/pods/a1c9d454-dbe9-4e6b-ab20-71b7388cedb0/volumes" Nov 03 22:43:01.167580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:01.167545 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:01.390160 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:01.390072 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:01.627423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:01.627155 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:03.790797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:03.790761 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:04.854511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:04.854481 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:43:04.855117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:04.855099 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:43:05.548537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:05.548486 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:05.764695 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:05.764661 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:05.986664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:05.982855 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:07.437521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.437489 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:07.646894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.646865 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:07.712699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.703843 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn"] Nov 03 22:43:07.714928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.714898 4298 cpu_manager.go:414] "containerMap: removing container" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" Nov 03 22:43:07.714928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.714927 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" Nov 03 22:43:07.715130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.714949 4298 cpu_manager.go:414] "containerMap: removing container" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" Nov 03 22:43:07.715130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.714960 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" Nov 03 22:43:07.715236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.715169 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" Nov 03 22:43:07.721741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.721717 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:07.756373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.756335 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf62853-8705-406f-a257-58c97f52df51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:07.756574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.756531 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf62853-8705-406f-a257-58c97f52df51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:07.756766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.756749 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24sp4\" (UniqueName: \"kubernetes.io/projected/1bf62853-8705-406f-a257-58c97f52df51-kube-api-access-24sp4\") pod \"maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:07.764268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.764240 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn"] Nov 03 22:43:07.807927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.807897 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:43:07.808309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.808278 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:43:07.858071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.857994 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf62853-8705-406f-a257-58c97f52df51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:07.858249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.858152 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf62853-8705-406f-a257-58c97f52df51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:07.858355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.858332 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24sp4\" (UniqueName: \"kubernetes.io/projected/1bf62853-8705-406f-a257-58c97f52df51-kube-api-access-24sp4\") pod \"maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:07.859244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.859212 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf62853-8705-406f-a257-58c97f52df51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:07.861924 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.861895 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf62853-8705-406f-a257-58c97f52df51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:07.917829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:07.917785 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24sp4\" (UniqueName: \"kubernetes.io/projected/1bf62853-8705-406f-a257-58c97f52df51-kube-api-access-24sp4\") pod \"maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:08.040009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.039910 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:08.041684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.041637 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8"] Nov 03 22:43:08.058691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.043031 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c9d454-dbe9-4e6b-ab20-71b7388cedb0" containerName="istio-proxy" Nov 03 22:43:08.066220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.066187 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8"] Nov 03 22:43:08.066372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.066279 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:43:08.120686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.119503 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-97965e\"" Nov 03 22:43:08.120686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.119912 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-97965e-dockercfg-vqcqc\"" Nov 03 22:43:08.150112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.150079 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn"] Nov 03 22:43:08.154756 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.154621 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:43:08.194085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.194051 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px87s\" (UniqueName: \"kubernetes.io/projected/1caea868-e30c-40ca-8b1c-2f82fef32b41-kube-api-access-px87s\") pod \"payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8\" (UID: \"1caea868-e30c-40ca-8b1c-2f82fef32b41\") " pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:43:08.194356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.194334 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caea868-e30c-40ca-8b1c-2f82fef32b41-plugins-config-volume\") pod \"payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8\" (UID: \"1caea868-e30c-40ca-8b1c-2f82fef32b41\") " pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:43:08.213168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.213069 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn"] Nov 03 22:43:08.295554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.295459 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwlxz\" (UniqueName: \"kubernetes.io/projected/4b6f0f1a-5eae-45e8-be70-d250125cb519-kube-api-access-vwlxz\") pod \"payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn\" (UID: \"4b6f0f1a-5eae-45e8-be70-d250125cb519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:43:08.295742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.295571 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px87s\" (UniqueName: \"kubernetes.io/projected/1caea868-e30c-40ca-8b1c-2f82fef32b41-kube-api-access-px87s\") pod \"payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8\" (UID: \"1caea868-e30c-40ca-8b1c-2f82fef32b41\") " pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:43:08.295742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.295656 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6f0f1a-5eae-45e8-be70-d250125cb519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn\" (UID: \"4b6f0f1a-5eae-45e8-be70-d250125cb519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:43:08.295918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.295883 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caea868-e30c-40ca-8b1c-2f82fef32b41-plugins-config-volume\") pod \"payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8\" (UID: \"1caea868-e30c-40ca-8b1c-2f82fef32b41\") " pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:43:08.296779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.296758 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caea868-e30c-40ca-8b1c-2f82fef32b41-plugins-config-volume\") pod \"payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8\" (UID: \"1caea868-e30c-40ca-8b1c-2f82fef32b41\") " pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:43:08.357532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.357506 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px87s\" (UniqueName: \"kubernetes.io/projected/1caea868-e30c-40ca-8b1c-2f82fef32b41-kube-api-access-px87s\") pod \"payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8\" (UID: \"1caea868-e30c-40ca-8b1c-2f82fef32b41\") " pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:43:08.397108 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.397072 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwlxz\" (UniqueName: \"kubernetes.io/projected/4b6f0f1a-5eae-45e8-be70-d250125cb519-kube-api-access-vwlxz\") pod \"payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn\" (UID: \"4b6f0f1a-5eae-45e8-be70-d250125cb519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:43:08.397276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.397163 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6f0f1a-5eae-45e8-be70-d250125cb519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn\" (UID: \"4b6f0f1a-5eae-45e8-be70-d250125cb519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:43:08.397875 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.397854 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6f0f1a-5eae-45e8-be70-d250125cb519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn\" (UID: \"4b6f0f1a-5eae-45e8-be70-d250125cb519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:43:08.419811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.419778 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:43:08.458121 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:43:08.458081 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf62853_8705_406f_a257_58c97f52df51.slice/crio-3b7c2ed0efe444ddefdceee0c5b9fa222666079296a6a8b8250c593dc7154a2b WatchSource:0}: Error finding container 3b7c2ed0efe444ddefdceee0c5b9fa222666079296a6a8b8250c593dc7154a2b: Status 404 returned error can't find the container with id 3b7c2ed0efe444ddefdceee0c5b9fa222666079296a6a8b8250c593dc7154a2b Nov 03 22:43:08.462385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.462358 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwlxz\" (UniqueName: \"kubernetes.io/projected/4b6f0f1a-5eae-45e8-be70-d250125cb519-kube-api-access-vwlxz\") pod \"payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn\" (UID: \"4b6f0f1a-5eae-45e8-be70-d250125cb519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:43:08.464825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.463659 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn"] Nov 03 22:43:08.471350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.471326 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:43:08.798003 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:43:08.797970 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caea868_e30c_40ca_8b1c_2f82fef32b41.slice/crio-17707d70541476271399aaddccaf9a34ff1a60fcf7dd7eebfa14fe1bff4a0a3c WatchSource:0}: Error finding container 17707d70541476271399aaddccaf9a34ff1a60fcf7dd7eebfa14fe1bff4a0a3c: Status 404 returned error can't find the container with id 17707d70541476271399aaddccaf9a34ff1a60fcf7dd7eebfa14fe1bff4a0a3c Nov 03 22:43:08.801306 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.801280 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8"] Nov 03 22:43:08.892077 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:43:08.892051 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6f0f1a_5eae_45e8_be70_d250125cb519.slice/crio-6f329cf749f24e2e19be7987eae69c7cb6b0c940688a1942808cb2ba217e8300 WatchSource:0}: Error finding container 6f329cf749f24e2e19be7987eae69c7cb6b0c940688a1942808cb2ba217e8300: Status 404 returned error can't find the container with id 6f329cf749f24e2e19be7987eae69c7cb6b0c940688a1942808cb2ba217e8300 Nov 03 22:43:08.898801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.898778 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn"] Nov 03 22:43:08.924415 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:08.924392 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:09.019819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:09.019782 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" event={"ID":"1bf62853-8705-406f-a257-58c97f52df51","Type":"ContainerStarted","Data":"936e875aded9b368624e81d37d7aaab7300b57e354fa79ffbaf2ea4bb0a78b61"} Nov 03 22:43:09.020088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:09.019842 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" event={"ID":"1bf62853-8705-406f-a257-58c97f52df51","Type":"ContainerStarted","Data":"3b7c2ed0efe444ddefdceee0c5b9fa222666079296a6a8b8250c593dc7154a2b"} Nov 03 22:43:09.021213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:09.021187 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" event={"ID":"4b6f0f1a-5eae-45e8-be70-d250125cb519","Type":"ContainerStarted","Data":"6f329cf749f24e2e19be7987eae69c7cb6b0c940688a1942808cb2ba217e8300"} Nov 03 22:43:09.022673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:09.022617 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" event={"ID":"1caea868-e30c-40ca-8b1c-2f82fef32b41","Type":"ContainerStarted","Data":"698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605"} Nov 03 22:43:09.022767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:09.022745 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" event={"ID":"1caea868-e30c-40ca-8b1c-2f82fef32b41","Type":"ContainerStarted","Data":"17707d70541476271399aaddccaf9a34ff1a60fcf7dd7eebfa14fe1bff4a0a3c"} Nov 03 22:43:09.022815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:09.022781 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:43:09.135316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:09.135022 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" podStartSLOduration=2.135004809 podStartE2EDuration="2.135004809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:09.133898465 +0000 UTC m=+2146.114632117" watchObservedRunningTime="2026-11-03 22:43:09.135004809 +0000 UTC m=+2146.115738464" Nov 03 22:43:09.254523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:09.254460 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" podStartSLOduration=2.254445867 podStartE2EDuration="2.254445867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:09.24994409 +0000 UTC m=+2146.230677742" watchObservedRunningTime="2026-11-03 22:43:09.254445867 +0000 UTC m=+2146.235179520" Nov 03 22:43:10.031034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:10.030994 4298 generic.go:356] "Generic (PLEG): container finished" podID="1bf62853-8705-406f-a257-58c97f52df51" containerID="936e875aded9b368624e81d37d7aaab7300b57e354fa79ffbaf2ea4bb0a78b61" exitCode=0 Nov 03 22:43:10.031496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:10.031080 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" event={"ID":"1bf62853-8705-406f-a257-58c97f52df51","Type":"ContainerDied","Data":"936e875aded9b368624e81d37d7aaab7300b57e354fa79ffbaf2ea4bb0a78b61"} Nov 03 22:43:10.032703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:10.032675 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" event={"ID":"4b6f0f1a-5eae-45e8-be70-d250125cb519","Type":"ContainerStarted","Data":"456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336"} Nov 03 22:43:10.032843 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:10.032829 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:43:10.261654 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:10.261584 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" podStartSLOduration=2.261569083 podStartE2EDuration="2.261569083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:10.258685128 +0000 UTC m=+2147.239418799" watchObservedRunningTime="2026-11-03 22:43:10.261569083 +0000 UTC m=+2147.242302763" Nov 03 22:43:11.200749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:11.200723 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:11.338982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:11.338899 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bf62853-8705-406f-a257-58c97f52df51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf62853-8705-406f-a257-58c97f52df51-maas-api-xks-ca\") pod \"1bf62853-8705-406f-a257-58c97f52df51\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " Nov 03 22:43:11.339163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:11.339020 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf62853-8705-406f-a257-58c97f52df51-kube-api-access-24sp4\" (UniqueName: \"kubernetes.io/projected/1bf62853-8705-406f-a257-58c97f52df51-kube-api-access-24sp4\") pod \"1bf62853-8705-406f-a257-58c97f52df51\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " Nov 03 22:43:11.339163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:11.339151 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf62853-8705-406f-a257-58c97f52df51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf62853-8705-406f-a257-58c97f52df51-maas-api-openshift-ca\") pod \"1bf62853-8705-406f-a257-58c97f52df51\" (UID: \"1bf62853-8705-406f-a257-58c97f52df51\") " Nov 03 22:43:11.339557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:11.339526 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf62853-8705-406f-a257-58c97f52df51-maas-api-openshift-ca" pod "1bf62853-8705-406f-a257-58c97f52df51" (UID: "1bf62853-8705-406f-a257-58c97f52df51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:11.339854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:11.339831 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf62853-8705-406f-a257-58c97f52df51-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.341044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:11.341011 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf62853-8705-406f-a257-58c97f52df51-maas-api-xks-ca" pod "1bf62853-8705-406f-a257-58c97f52df51" (UID: "1bf62853-8705-406f-a257-58c97f52df51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:11.341138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:11.341121 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf62853-8705-406f-a257-58c97f52df51-kube-api-access-24sp4" pod "1bf62853-8705-406f-a257-58c97f52df51" (UID: "1bf62853-8705-406f-a257-58c97f52df51"). InnerVolumeSpecName "kube-api-access-24sp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:11.440967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:11.440935 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24sp4\" (UniqueName: \"kubernetes.io/projected/1bf62853-8705-406f-a257-58c97f52df51-kube-api-access-24sp4\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.440967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:11.440963 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf62853-8705-406f-a257-58c97f52df51-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:12.048022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:12.047988 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" event={"ID":"1bf62853-8705-406f-a257-58c97f52df51","Type":"ContainerDied","Data":"3b7c2ed0efe444ddefdceee0c5b9fa222666079296a6a8b8250c593dc7154a2b"} Nov 03 22:43:12.048022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:12.048018 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn" Nov 03 22:43:12.048242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:12.048036 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b7c2ed0efe444ddefdceee0c5b9fa222666079296a6a8b8250c593dc7154a2b" Nov 03 22:43:13.909703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:13.909657 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:15.479668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:15.479580 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:17.553599 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:43:17.553561 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:43:18.957781 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:18.957732 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:19.573828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:19.573797 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4"] Nov 03 22:43:19.694565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:19.694531 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v"] Nov 03 22:43:20.034666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:20.033802 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:20.035803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:20.035768 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:43:20.303737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:20.303622 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:21.043303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:21.043278 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:43:24.025081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:24.025017 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:24.574823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:24.574786 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" podUID="59bce71a-9625-41ee-91f7-f361e31eb9c0" containerName="payload-processing" containerID="cri-o://64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029" gracePeriod=25 Nov 03 22:43:24.695597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:24.695560 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" podUID="0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c" containerName="payload-pre-processing" containerID="cri-o://410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b" gracePeriod=25 Nov 03 22:43:24.865736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:24.865710 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:43:24.962370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:24.962349 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59bce71a-9625-41ee-91f7-f361e31eb9c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59bce71a-9625-41ee-91f7-f361e31eb9c0-plugins-config-volume\") pod \"59bce71a-9625-41ee-91f7-f361e31eb9c0\" (UID: \"59bce71a-9625-41ee-91f7-f361e31eb9c0\") " Nov 03 22:43:24.962446 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:24.962391 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bce71a-9625-41ee-91f7-f361e31eb9c0-kube-api-access-ctlvq\" (UniqueName: \"kubernetes.io/projected/59bce71a-9625-41ee-91f7-f361e31eb9c0-kube-api-access-ctlvq\") pod \"59bce71a-9625-41ee-91f7-f361e31eb9c0\" (UID: \"59bce71a-9625-41ee-91f7-f361e31eb9c0\") " Nov 03 22:43:24.962850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:24.962820 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59bce71a-9625-41ee-91f7-f361e31eb9c0-plugins-config-volume" pod "59bce71a-9625-41ee-91f7-f361e31eb9c0" (UID: "59bce71a-9625-41ee-91f7-f361e31eb9c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:24.964682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:24.964629 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bce71a-9625-41ee-91f7-f361e31eb9c0-kube-api-access-ctlvq" pod "59bce71a-9625-41ee-91f7-f361e31eb9c0" (UID: "59bce71a-9625-41ee-91f7-f361e31eb9c0"). InnerVolumeSpecName "kube-api-access-ctlvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:24.974600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:24.974574 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:43:25.063420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.063388 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-plugins-config-volume\") pod \"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c\" (UID: \"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c\") " Nov 03 22:43:25.063854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.063435 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-kube-api-access-rzrm4\" (UniqueName: \"kubernetes.io/projected/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-kube-api-access-rzrm4\") pod \"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c\" (UID: \"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c\") " Nov 03 22:43:25.063854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.063756 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-plugins-config-volume" pod "0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c" (UID: "0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:25.064073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.064056 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:25.064118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.064077 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59bce71a-9625-41ee-91f7-f361e31eb9c0-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:25.064118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.064086 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctlvq\" (UniqueName: \"kubernetes.io/projected/59bce71a-9625-41ee-91f7-f361e31eb9c0-kube-api-access-ctlvq\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:25.065499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.065478 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-kube-api-access-rzrm4" pod "0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c" (UID: "0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c"). InnerVolumeSpecName "kube-api-access-rzrm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:25.165172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.165136 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzrm4\" (UniqueName: \"kubernetes.io/projected/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c-kube-api-access-rzrm4\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:25.310953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.310904 4298 generic.go:356] "Generic (PLEG): container finished" podID="0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c" containerID="410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b" exitCode=0 Nov 03 22:43:25.311148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.311102 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" event={"ID":"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c","Type":"ContainerDied","Data":"410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b"} Nov 03 22:43:25.311208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.311189 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" event={"ID":"0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c","Type":"ContainerDied","Data":"78163ef30f4ebe2b53f88f25a82cc1d6e8ed487b0b716f596446270b93038364"} Nov 03 22:43:25.311258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.311217 4298 scope.go:122] "RemoveContainer" containerID="410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b" Nov 03 22:43:25.311605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.311386 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v" Nov 03 22:43:25.319022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.318981 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" Nov 03 22:43:25.319375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.319346 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" event={"ID":"59bce71a-9625-41ee-91f7-f361e31eb9c0","Type":"ContainerDied","Data":"64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029"} Nov 03 22:43:25.319667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.319624 4298 generic.go:356] "Generic (PLEG): container finished" podID="59bce71a-9625-41ee-91f7-f361e31eb9c0" containerID="64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029" exitCode=0 Nov 03 22:43:25.319783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.319756 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4" event={"ID":"59bce71a-9625-41ee-91f7-f361e31eb9c0","Type":"ContainerDied","Data":"22c8c1002dbc01a97ace36749aabc6d945531b1b48531a851968d52364c123cd"} Nov 03 22:43:25.328351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.328334 4298 scope.go:122] "RemoveContainer" containerID="410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b" Nov 03 22:43:25.328712 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:43:25.328686 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b\": container with ID starting with 410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b not found: ID does not exist" containerID="410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b" Nov 03 22:43:25.328809 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.328721 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b"} err="failed to get container status \"410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b\": rpc error: code = NotFound desc = could not find container \"410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b\": container with ID starting with 410e74fd571378e34b0f263755c21ad72ebea5b88999bfe84096a6a9adcc1e8b not found: ID does not exist" Nov 03 22:43:25.328809 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.328738 4298 scope.go:122] "RemoveContainer" containerID="64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029" Nov 03 22:43:25.338820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.338802 4298 scope.go:122] "RemoveContainer" containerID="64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029" Nov 03 22:43:25.339082 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:43:25.339064 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029\": container with ID starting with 64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029 not found: ID does not exist" containerID="64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029" Nov 03 22:43:25.339139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.339091 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029"} err="failed to get container status \"64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029\": rpc error: code = NotFound desc = could not find container \"64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029\": container with ID starting with 64338b284d61c2cee270f902f5698cded0506698b5a0f5887926ffb1f7434029 not found: ID does not exist" Nov 03 22:43:25.495517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.495481 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v"] Nov 03 22:43:25.545590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.545558 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9ef23-6468568f8c-89z2v"] Nov 03 22:43:25.614656 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.614606 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:25.616345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.616324 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4"] Nov 03 22:43:25.621284 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.621259 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9ef23-f579c7c6-f8gc4"] Nov 03 22:43:25.807515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:25.807428 4298 scope.go:122] "RemoveContainer" containerID="37f3f4eeb46ff68096d4edfe42134ade9b229a34e94d29b18170ffe9be9b578f" Nov 03 22:43:27.156821 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:27.156790 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c" path="/var/lib/kubelet/pods/0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c/volumes" Nov 03 22:43:27.157328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:27.157312 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59bce71a-9625-41ee-91f7-f361e31eb9c0" path="/var/lib/kubelet/pods/59bce71a-9625-41ee-91f7-f361e31eb9c0/volumes" Nov 03 22:43:29.071874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:29.071813 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:33.277544 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:33.277500 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:35.186415 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:35.186328 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:35.519297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:35.519263 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:35.783885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:35.783809 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:35.962615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:35.962584 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9"] Nov 03 22:43:36.012440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:36.012411 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc88eb47-xctj9"] Nov 03 22:43:36.393343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:36.393307 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:36.445654 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:36.445615 4298 generic.go:356] "Generic (PLEG): container finished" podID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerID="9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1" exitCode=0 Nov 03 22:43:36.445840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:36.445702 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" event={"ID":"a85d19cf-d09f-427c-a04c-f97d95723d99","Type":"ContainerDied","Data":"9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1"} Nov 03 22:43:36.446150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:36.446131 4298 scope.go:122] "RemoveContainer" containerID="9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1" Nov 03 22:43:36.447774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:36.447748 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:43:36.447855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:36.447815 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:43:36.447855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:36.447845 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:43:37.136759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:37.136729 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d150f82-ce58-454e-b2a6-cd3abff079e4" path="/var/lib/kubelet/pods/0d150f82-ce58-454e-b2a6-cd3abff079e4/volumes" Nov 03 22:43:37.462069 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:37.462022 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" event={"ID":"a85d19cf-d09f-427c-a04c-f97d95723d99","Type":"ContainerStarted","Data":"8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594"} Nov 03 22:43:38.161207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:38.161169 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:43:38.161439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:38.161395 4298 patch_prober.go:29] interesting pod/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 22:43:38.161501 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:38.161459 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 22:43:39.161206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:39.161176 4298 patch_prober.go:29] interesting pod/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 22:43:39.161599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:39.161226 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 22:43:39.953782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:39.953748 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:40.146667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:40.145311 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:40.163684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:40.163632 4298 patch_prober.go:29] interesting pod/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 22:43:40.164302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:40.164268 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 22:43:40.340223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:40.340130 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:41.161915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:41.161883 4298 patch_prober.go:29] interesting pod/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 22:43:41.162098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:41.161931 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 22:43:41.421730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:41.421548 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:42.161440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:42.161391 4298 patch_prober.go:29] interesting pod/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 22:43:42.161682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:42.161470 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 22:43:43.052328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:43.052295 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm"] Nov 03 22:43:43.102349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:43.102311 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48738db1-5pljm"] Nov 03 22:43:43.138014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:43.137972 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c92f85-e307-4ca6-8e7e-6089d34b3a50" path="/var/lib/kubelet/pods/40c92f85-e307-4ca6-8e7e-6089d34b3a50/volumes" Nov 03 22:43:43.227513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:43.227475 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:43:43.227861 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:43.227828 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:43:43.228944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:43.228924 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:43:45.099728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:45.099691 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:47.552419 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:43:47.552371 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:43:49.648828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:49.648791 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:49.863636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:49.863590 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:50.062796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:50.062764 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:50.874669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:50.874612 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:51.127308 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:51.127215 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:54.357371 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:54.356837 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:58.835554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:58.835517 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:59.016476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:59.016444 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:59.418948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:43:59.418912 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:03.576734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:03.576623 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:08.099184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:08.099148 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:12.685729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:12.685684 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:15.820790 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:15.820740 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:17.054856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:17.054818 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:17.234112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:17.234074 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:17.486268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:17.486230 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:17.557504 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:44:17.557451 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:44:17.663586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:17.663502 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:18.030532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:18.030500 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:18.962638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:18.962603 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:20.431331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:20.431296 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:20.925130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:20.925101 4298 generic.go:356] "Generic (PLEG): container finished" podID="a535c385-80f0-49ec-870d-7d83ccce3073" containerID="989c34827f115cc66afea8a033f30be9c8c816375483c25696dec12d6b8af6d0" exitCode=0 Nov 03 22:44:20.925330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:20.925176 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" event={"ID":"a535c385-80f0-49ec-870d-7d83ccce3073","Type":"ContainerDied","Data":"989c34827f115cc66afea8a033f30be9c8c816375483c25696dec12d6b8af6d0"} Nov 03 22:44:20.925741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:20.925721 4298 scope.go:122] "RemoveContainer" containerID="989c34827f115cc66afea8a033f30be9c8c816375483c25696dec12d6b8af6d0" Nov 03 22:44:20.927473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:20.927446 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:44:20.927552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:20.927518 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:44:20.927590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:20.927554 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:44:21.773027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:21.772995 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:21.955282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:21.955229 4298 generic.go:356] "Generic (PLEG): container finished" podID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerID="cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55" exitCode=0 Nov 03 22:44:21.955483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:21.955297 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" event={"ID":"5659da87-ec7e-4434-a21c-c33ac6aacd20","Type":"ContainerDied","Data":"cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55"} Nov 03 22:44:21.955853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:21.955833 4298 scope.go:122] "RemoveContainer" containerID="cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55" Nov 03 22:44:21.957615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:21.957593 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" event={"ID":"a535c385-80f0-49ec-870d-7d83ccce3073","Type":"ContainerStarted","Data":"35b258bf91f6ea199c1dfe0531e6c8bacd869f177b6ed2979fff1c2fecbbf1e5"} Nov 03 22:44:21.957826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:21.957798 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:44:21.957915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:21.957886 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:44:21.957979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:21.957942 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:44:22.259736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:22.259223 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:44:22.259736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:22.259343 4298 patch_prober.go:29] interesting pod/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:44:22.259736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:22.259398 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:44:23.022824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:23.022791 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" event={"ID":"5659da87-ec7e-4434-a21c-c33ac6aacd20","Type":"ContainerStarted","Data":"217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840"} Nov 03 22:44:23.260665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:23.260104 4298 patch_prober.go:29] interesting pod/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:44:23.260665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:23.260175 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:44:23.973668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:23.972806 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:44:23.976691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:23.974736 4298 patch_prober.go:29] interesting pod/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:44:23.976691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:23.974787 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 22:44:24.259148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:24.259054 4298 patch_prober.go:29] interesting pod/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:44:24.259723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:24.259689 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:44:24.787067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:24.787037 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:24.965675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:24.965104 4298 patch_prober.go:29] interesting pod/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:44:24.965675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:24.965185 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 22:44:24.974441 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:24.974306 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:25.108721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:25.104240 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl"] Nov 03 22:44:25.108721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:25.104478 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" podUID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" containerName="istio-proxy" containerID="cri-o://de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256" gracePeriod=30 Nov 03 22:44:25.113474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:25.112988 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm"] Nov 03 22:44:25.113474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:25.113215 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" podUID="52c39859-96fe-412f-8cc9-63b7c5f29bea" containerName="istio-proxy" containerID="cri-o://4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8" gracePeriod=30 Nov 03 22:44:25.211583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:25.195819 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:25.258907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:25.258873 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw"] Nov 03 22:44:25.318664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:25.294789 4298 patch_prober.go:29] interesting pod/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:44:25.318664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:25.310803 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:44:25.318664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:25.310922 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerName="istio-proxy" containerID="cri-o://5adf3240ec844e840104c47ed1dad3d2778a35c4584665b06231ffdec5373ac1" gracePeriod=30 Nov 03 22:44:25.478666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:25.476096 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:44:25.779698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:25.778613 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:26.014319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:26.014240 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:26.088778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:26.069483 4298 patch_prober.go:29] interesting pod/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:44:26.088778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:26.069561 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 22:44:26.337673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:26.322745 4298 patch_prober.go:29] interesting pod/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:44:26.337673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:26.322806 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:44:26.964581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:26.964538 4298 patch_prober.go:29] interesting pod/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:44:26.964910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:26.964877 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 22:44:27.272725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:27.266079 4298 patch_prober.go:29] interesting pod/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:44:27.272725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:27.266144 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:44:27.672023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:27.671926 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:27.855669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:27.854146 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:27.919435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:27.919406 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn"] Nov 03 22:44:27.957575 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:27.957542 4298 patch_prober.go:29] interesting pod/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:44:27.957806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:27.957607 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 22:44:27.969431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:27.969400 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9ef23-f9sqn"] Nov 03 22:44:28.087015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.086984 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:28.263298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.263218 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:44:28.263532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.263511 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:44:28.264101 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.264085 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:44:28.363323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.363291 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:28.572720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.572341 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm"] Nov 03 22:44:28.575514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.573556 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1bf62853-8705-406f-a257-58c97f52df51" containerName="revoke-keys" Nov 03 22:44:28.575514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.573582 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf62853-8705-406f-a257-58c97f52df51" containerName="revoke-keys" Nov 03 22:44:28.575514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.573605 4298 cpu_manager.go:414] "containerMap: removing container" podUID="59bce71a-9625-41ee-91f7-f361e31eb9c0" containerName="payload-processing" Nov 03 22:44:28.575514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.573613 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bce71a-9625-41ee-91f7-f361e31eb9c0" containerName="payload-processing" Nov 03 22:44:28.575514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.573670 4298 cpu_manager.go:414] "containerMap: removing container" podUID="0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c" containerName="payload-pre-processing" Nov 03 22:44:28.575514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.573678 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c" containerName="payload-pre-processing" Nov 03 22:44:28.575514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.573835 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf62853-8705-406f-a257-58c97f52df51" containerName="revoke-keys" Nov 03 22:44:28.575514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.573847 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bce71a-9625-41ee-91f7-f361e31eb9c0" containerName="payload-processing" Nov 03 22:44:28.575514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.573862 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9ffd2c-3fe3-42fc-a0cf-a7486b8a148c" containerName="payload-pre-processing" Nov 03 22:44:28.581233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.581187 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.615950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.615912 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-podinfo\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.616147 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.615966 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-envoy\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.616147 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.616117 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcg6l\" (UniqueName: \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-kube-api-access-pcg6l\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.616272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.616178 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74935ed0-dfaa-4277-af6d-76b89f6e8970-istiod-ca-cert\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.616340 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.616280 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-credential-socket\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.616461 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.616432 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-data\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.616521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.616495 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-token\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.616619 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.616598 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-certs\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.616731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.616708 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-socket\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.631473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.631431 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm"] Nov 03 22:44:28.637771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.637739 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-01bb3a-openshift-default-dockercfg-bctss\"" Nov 03 22:44:28.681023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.680980 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:28.718932 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.718890 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-podinfo\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.720770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.720744 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-envoy\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.721339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.721321 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcg6l\" (UniqueName: \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-kube-api-access-pcg6l\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.721848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.721829 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74935ed0-dfaa-4277-af6d-76b89f6e8970-istiod-ca-cert\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.722858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.722727 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74935ed0-dfaa-4277-af6d-76b89f6e8970-istiod-ca-cert\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.723020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.723000 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-credential-socket\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.723494 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.723379 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-credential-socket\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.778663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.734522 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-data\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.778663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.734585 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-token\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.778663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.734682 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-certs\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.778663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.734793 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-socket\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.778663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.735437 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-socket\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.778663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.756492 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-podinfo\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.778663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.757498 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-data\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.778663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.757759 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-certs\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.800982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.797038 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-envoy\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.833851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.819053 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcg6l\" (UniqueName: \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-kube-api-access-pcg6l\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.833851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.820596 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-token\") pod \"e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:28.901553 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:28.900091 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:29.114343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.114219 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:44:29.115698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.115674 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:44:29.136088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.136057 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf62853-8705-406f-a257-58c97f52df51" path="/var/lib/kubelet/pods/1bf62853-8705-406f-a257-58c97f52df51/volumes" Nov 03 22:44:29.209892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.209864 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:29.254928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.254897 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:44:29.255089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.254951 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" podUID="52c39859-96fe-412f-8cc9-63b7c5f29bea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:44:29.392807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.392705 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm"] Nov 03 22:44:29.413892 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:44:29.413853 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74935ed0_dfaa_4277_af6d_76b89f6e8970.slice/crio-7eacb88399266cfbded59d152629e4d28e70d4082eea932b2ef9d96a0750dd5a WatchSource:0}: Error finding container 7eacb88399266cfbded59d152629e4d28e70d4082eea932b2ef9d96a0750dd5a: Status 404 returned error can't find the container with id 7eacb88399266cfbded59d152629e4d28e70d4082eea932b2ef9d96a0750dd5a Nov 03 22:44:29.417241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.417205 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:44:29.417342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.417299 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:44:29.417396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.417349 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:44:29.486116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.486084 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" event={"ID":"74935ed0-dfaa-4277-af6d-76b89f6e8970","Type":"ContainerStarted","Data":"7eacb88399266cfbded59d152629e4d28e70d4082eea932b2ef9d96a0750dd5a"} Nov 03 22:44:29.565004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.564961 4298 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:44:29.565168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.565020 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" podUID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:44:29.613017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.612986 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:29.797417 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:29.797381 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:30.487060 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.487034 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:44:30.491729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.491702 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:44:30.504470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.504355 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" event={"ID":"74935ed0-dfaa-4277-af6d-76b89f6e8970","Type":"ContainerStarted","Data":"a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c"} Nov 03 22:44:30.506219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.506179 4298 generic.go:356] "Generic (PLEG): container finished" podID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" containerID="de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256" exitCode=0 Nov 03 22:44:30.506357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.506309 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" event={"ID":"84461fb7-32f0-48ea-b8dd-f7a03d6b3191","Type":"ContainerDied","Data":"de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256"} Nov 03 22:44:30.506427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.506353 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" event={"ID":"84461fb7-32f0-48ea-b8dd-f7a03d6b3191","Type":"ContainerDied","Data":"f1cd5641c7d99aa8184d0beb2e49f89bcdd824858589101e260fa7600d05830e"} Nov 03 22:44:30.506427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.506382 4298 scope.go:122] "RemoveContainer" containerID="de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256" Nov 03 22:44:30.506541 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.506526 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl" Nov 03 22:44:30.509308 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.509249 4298 generic.go:356] "Generic (PLEG): container finished" podID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerID="5adf3240ec844e840104c47ed1dad3d2778a35c4584665b06231ffdec5373ac1" exitCode=0 Nov 03 22:44:30.509467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.509351 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" event={"ID":"5f6549bc-c506-425a-bf96-9d3ba30b9d6c","Type":"ContainerDied","Data":"5adf3240ec844e840104c47ed1dad3d2778a35c4584665b06231ffdec5373ac1"} Nov 03 22:44:30.519170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.519142 4298 generic.go:356] "Generic (PLEG): container finished" podID="52c39859-96fe-412f-8cc9-63b7c5f29bea" containerID="4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8" exitCode=0 Nov 03 22:44:30.519282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.519248 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" event={"ID":"52c39859-96fe-412f-8cc9-63b7c5f29bea","Type":"ContainerDied","Data":"4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8"} Nov 03 22:44:30.519346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.519311 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" event={"ID":"52c39859-96fe-412f-8cc9-63b7c5f29bea","Type":"ContainerDied","Data":"0b45ee90090f3ac053541c2d06af1f4d12425a014bbd63710fd5309589bf6820"} Nov 03 22:44:30.519417 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.519406 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm" Nov 03 22:44:30.523012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.522993 4298 scope.go:122] "RemoveContainer" containerID="de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256" Nov 03 22:44:30.523388 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:44:30.523366 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256\": container with ID starting with de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256 not found: ID does not exist" containerID="de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256" Nov 03 22:44:30.523453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.523401 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256"} err="failed to get container status \"de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256\": rpc error: code = NotFound desc = could not find container \"de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256\": container with ID starting with de992e86545cff278d7cf4baf308e8bb8dce498e282ef81d5a7507712b855256 not found: ID does not exist" Nov 03 22:44:30.523453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.523424 4298 scope.go:122] "RemoveContainer" containerID="4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8" Nov 03 22:44:30.536672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.536634 4298 scope.go:122] "RemoveContainer" containerID="4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8" Nov 03 22:44:30.536945 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:44:30.536918 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8\": container with ID starting with 4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8 not found: ID does not exist" containerID="4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8" Nov 03 22:44:30.537022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.536953 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8"} err="failed to get container status \"4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8\": rpc error: code = NotFound desc = could not find container \"4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8\": container with ID starting with 4ac328dd44ed0a1d9cdf3fa7c59f62b660209202e8e065aaf76980619eca79e8 not found: ID does not exist" Nov 03 22:44:30.565899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.565865 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-socket\") pod \"52c39859-96fe-412f-8cc9-63b7c5f29bea\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " Nov 03 22:44:30.566066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.565934 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istiod-ca-cert\") pod \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " Nov 03 22:44:30.566066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.566002 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c39859-96fe-412f-8cc9-63b7c5f29bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c39859-96fe-412f-8cc9-63b7c5f29bea-istiod-ca-cert\") pod \"52c39859-96fe-412f-8cc9-63b7c5f29bea\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " Nov 03 22:44:30.566066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.566051 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-socket\") pod \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " Nov 03 22:44:30.566259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.566169 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-socket" pod "52c39859-96fe-412f-8cc9-63b7c5f29bea" (UID: "52c39859-96fe-412f-8cc9-63b7c5f29bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.566401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.566376 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-socket" pod "84461fb7-32f0-48ea-b8dd-f7a03d6b3191" (UID: "84461fb7-32f0-48ea-b8dd-f7a03d6b3191"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.566401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.566396 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istiod-ca-cert" pod "84461fb7-32f0-48ea-b8dd-f7a03d6b3191" (UID: "84461fb7-32f0-48ea-b8dd-f7a03d6b3191"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.566865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.566781 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c39859-96fe-412f-8cc9-63b7c5f29bea-istiod-ca-cert" pod "52c39859-96fe-412f-8cc9-63b7c5f29bea" (UID: "52c39859-96fe-412f-8cc9-63b7c5f29bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.566865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.566851 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-token\" (UniqueName: \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-token\") pod \"52c39859-96fe-412f-8cc9-63b7c5f29bea\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " Nov 03 22:44:30.566995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.566900 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-certs\") pod \"52c39859-96fe-412f-8cc9-63b7c5f29bea\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " Nov 03 22:44:30.566995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.566961 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-kube-api-access-kvggm\" (UniqueName: \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-kube-api-access-kvggm\") pod \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " Nov 03 22:44:30.567150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.567012 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-token\" (UniqueName: \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-token\") pod \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " Nov 03 22:44:30.567150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.567048 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-podinfo\") pod \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " Nov 03 22:44:30.567150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.567100 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-envoy\") pod \"52c39859-96fe-412f-8cc9-63b7c5f29bea\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " Nov 03 22:44:30.567318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.567178 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-certs" pod "52c39859-96fe-412f-8cc9-63b7c5f29bea" (UID: "52c39859-96fe-412f-8cc9-63b7c5f29bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.567318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.567237 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-envoy\") pod \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " Nov 03 22:44:30.567423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.567317 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-podinfo\") pod \"52c39859-96fe-412f-8cc9-63b7c5f29bea\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " Nov 03 22:44:30.567480 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.567423 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-certs\") pod \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " Nov 03 22:44:30.568271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.568244 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-credential-socket\") pod \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " Nov 03 22:44:30.568392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.568297 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-credential-socket\") pod \"52c39859-96fe-412f-8cc9-63b7c5f29bea\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " Nov 03 22:44:30.568392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.568344 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-data\") pod \"52c39859-96fe-412f-8cc9-63b7c5f29bea\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " Nov 03 22:44:30.568514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.568394 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-kube-api-access-mpvsg\" (UniqueName: \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-kube-api-access-mpvsg\") pod \"52c39859-96fe-412f-8cc9-63b7c5f29bea\" (UID: \"52c39859-96fe-412f-8cc9-63b7c5f29bea\") " Nov 03 22:44:30.568514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.568430 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-data\") pod \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\" (UID: \"84461fb7-32f0-48ea-b8dd-f7a03d6b3191\") " Nov 03 22:44:30.569259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.569232 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.569344 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.569327 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.569418 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.569345 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c39859-96fe-412f-8cc9-63b7c5f29bea-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.569418 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.569360 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.569418 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.569377 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.569632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.569587 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-token" pod "52c39859-96fe-412f-8cc9-63b7c5f29bea" (UID: "52c39859-96fe-412f-8cc9-63b7c5f29bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.569942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.569877 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-certs" pod "84461fb7-32f0-48ea-b8dd-f7a03d6b3191" (UID: "84461fb7-32f0-48ea-b8dd-f7a03d6b3191"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.570080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.570058 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-data" pod "84461fb7-32f0-48ea-b8dd-f7a03d6b3191" (UID: "84461fb7-32f0-48ea-b8dd-f7a03d6b3191"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.571376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.571351 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-credential-socket" pod "52c39859-96fe-412f-8cc9-63b7c5f29bea" (UID: "52c39859-96fe-412f-8cc9-63b7c5f29bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.571557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.571534 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-credential-socket" pod "84461fb7-32f0-48ea-b8dd-f7a03d6b3191" (UID: "84461fb7-32f0-48ea-b8dd-f7a03d6b3191"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.571689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.571633 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-kube-api-access-mpvsg" pod "52c39859-96fe-412f-8cc9-63b7c5f29bea" (UID: "52c39859-96fe-412f-8cc9-63b7c5f29bea"). InnerVolumeSpecName "kube-api-access-mpvsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.571980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.571889 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-data" pod "52c39859-96fe-412f-8cc9-63b7c5f29bea" (UID: "52c39859-96fe-412f-8cc9-63b7c5f29bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.572157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.572093 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-token" pod "84461fb7-32f0-48ea-b8dd-f7a03d6b3191" (UID: "84461fb7-32f0-48ea-b8dd-f7a03d6b3191"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.573523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.573502 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-envoy" pod "52c39859-96fe-412f-8cc9-63b7c5f29bea" (UID: "52c39859-96fe-412f-8cc9-63b7c5f29bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.573613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.573517 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-podinfo" pod "84461fb7-32f0-48ea-b8dd-f7a03d6b3191" (UID: "84461fb7-32f0-48ea-b8dd-f7a03d6b3191"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:30.573613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.573529 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-envoy" pod "84461fb7-32f0-48ea-b8dd-f7a03d6b3191" (UID: "84461fb7-32f0-48ea-b8dd-f7a03d6b3191"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.573731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.573661 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-podinfo" pod "52c39859-96fe-412f-8cc9-63b7c5f29bea" (UID: "52c39859-96fe-412f-8cc9-63b7c5f29bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:30.573953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.573933 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-kube-api-access-kvggm" pod "84461fb7-32f0-48ea-b8dd-f7a03d6b3191" (UID: "84461fb7-32f0-48ea-b8dd-f7a03d6b3191"). InnerVolumeSpecName "kube-api-access-kvggm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.581464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.581445 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:44:30.671832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.671748 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-token\" (UniqueName: \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-token\") pod \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " Nov 03 22:44:30.671987 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.671837 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-podinfo\") pod \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " Nov 03 22:44:30.671987 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.671895 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istiod-ca-cert\") pod \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " Nov 03 22:44:30.672077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.672052 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-credential-socket\") pod \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " Nov 03 22:44:30.672133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.672097 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-data\") pod \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " Nov 03 22:44:30.672258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.672141 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-envoy\") pod \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " Nov 03 22:44:30.672258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.672193 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-socket\") pod \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " Nov 03 22:44:30.672389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.672267 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-certs\") pod \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " Nov 03 22:44:30.672389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.672331 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-kube-api-access-vpmh9\" (UniqueName: \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-kube-api-access-vpmh9\") pod \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\" (UID: \"5f6549bc-c506-425a-bf96-9d3ba30b9d6c\") " Nov 03 22:44:30.672554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.672530 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istiod-ca-cert" pod "5f6549bc-c506-425a-bf96-9d3ba30b9d6c" (UID: "5f6549bc-c506-425a-bf96-9d3ba30b9d6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.673228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673067 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-credential-socket" pod "5f6549bc-c506-425a-bf96-9d3ba30b9d6c" (UID: "5f6549bc-c506-425a-bf96-9d3ba30b9d6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.673228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673083 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-certs" pod "5f6549bc-c506-425a-bf96-9d3ba30b9d6c" (UID: "5f6549bc-c506-425a-bf96-9d3ba30b9d6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.673228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673192 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673222 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvggm\" (UniqueName: \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-kube-api-access-kvggm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673239 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673258 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673272 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673287 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673301 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673315 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673332 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673346 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673360 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c39859-96fe-412f-8cc9-63b7c5f29bea-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673366 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-socket" pod "5f6549bc-c506-425a-bf96-9d3ba30b9d6c" (UID: "5f6549bc-c506-425a-bf96-9d3ba30b9d6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673375 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpvsg\" (UniqueName: \"kubernetes.io/projected/52c39859-96fe-412f-8cc9-63b7c5f29bea-kube-api-access-mpvsg\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673411 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84461fb7-32f0-48ea-b8dd-f7a03d6b3191-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.673515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.673427 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.674432 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.674401 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-token" pod "5f6549bc-c506-425a-bf96-9d3ba30b9d6c" (UID: "5f6549bc-c506-425a-bf96-9d3ba30b9d6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.674561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.674442 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-data" pod "5f6549bc-c506-425a-bf96-9d3ba30b9d6c" (UID: "5f6549bc-c506-425a-bf96-9d3ba30b9d6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.674819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.674799 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-envoy" pod "5f6549bc-c506-425a-bf96-9d3ba30b9d6c" (UID: "5f6549bc-c506-425a-bf96-9d3ba30b9d6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.674896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.674853 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-podinfo" pod "5f6549bc-c506-425a-bf96-9d3ba30b9d6c" (UID: "5f6549bc-c506-425a-bf96-9d3ba30b9d6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:30.676348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.676327 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-kube-api-access-vpmh9" pod "5f6549bc-c506-425a-bf96-9d3ba30b9d6c" (UID: "5f6549bc-c506-425a-bf96-9d3ba30b9d6c"). InnerVolumeSpecName "kube-api-access-vpmh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.774448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.774414 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.774448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.774446 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpmh9\" (UniqueName: \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-kube-api-access-vpmh9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.774448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.774457 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.774717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.774466 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.774717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.774474 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.774717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.774483 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.774717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.774491 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.774717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.774498 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6549bc-c506-425a-bf96-9d3ba30b9d6c-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.778369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.778317 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" podStartSLOduration=2.778304597 podStartE2EDuration="2.778304597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:30.775872483 +0000 UTC m=+2227.756606137" watchObservedRunningTime="2026-11-03 22:44:30.778304597 +0000 UTC m=+2227.759038327" Nov 03 22:44:30.852822 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:44:30.852783 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c39859_96fe_412f_8cc9_63b7c5f29bea.slice/crio-0b45ee90090f3ac053541c2d06af1f4d12425a014bbd63710fd5309589bf6820\": RecentStats: unable to find data in memory cache]" Nov 03 22:44:30.909685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.909404 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:30.915815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:30.915781 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:31.296356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:31.296324 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm"] Nov 03 22:44:31.301159 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:31.301138 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8f2zm"] Nov 03 22:44:31.410958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:31.410873 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl"] Nov 03 22:44:31.415881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:31.415853 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mqnpl"] Nov 03 22:44:31.528713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:31.528673 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" event={"ID":"5f6549bc-c506-425a-bf96-9d3ba30b9d6c","Type":"ContainerDied","Data":"a793929ac7321b47166eb3b1f55e76b286a8eb53c78a0587d212e295a13bc56a"} Nov 03 22:44:31.528713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:31.528699 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw" Nov 03 22:44:31.528915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:31.528746 4298 scope.go:122] "RemoveContainer" containerID="5adf3240ec844e840104c47ed1dad3d2778a35c4584665b06231ffdec5373ac1" Nov 03 22:44:31.530452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:31.530428 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:31.531693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:31.531673 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:44:31.817761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:31.817731 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw"] Nov 03 22:44:31.822836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:31.822810 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-wnbrw"] Nov 03 22:44:32.098965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.098887 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj"] Nov 03 22:44:32.099517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.099503 4298 cpu_manager.go:414] "containerMap: removing container" podUID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" containerName="istio-proxy" Nov 03 22:44:32.099576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.099517 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" containerName="istio-proxy" Nov 03 22:44:32.099576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.099534 4298 cpu_manager.go:414] "containerMap: removing container" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerName="istio-proxy" Nov 03 22:44:32.099576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.099540 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerName="istio-proxy" Nov 03 22:44:32.099576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.099567 4298 cpu_manager.go:414] "containerMap: removing container" podUID="52c39859-96fe-412f-8cc9-63b7c5f29bea" containerName="istio-proxy" Nov 03 22:44:32.099576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.099573 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c39859-96fe-412f-8cc9-63b7c5f29bea" containerName="istio-proxy" Nov 03 22:44:32.099756 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.099680 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c39859-96fe-412f-8cc9-63b7c5f29bea" containerName="istio-proxy" Nov 03 22:44:32.099756 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.099692 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" containerName="istio-proxy" Nov 03 22:44:32.099756 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.099698 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" containerName="istio-proxy" Nov 03 22:44:32.104503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.104487 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:32.158134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.158068 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:44:32.158134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.158114 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:44:32.160627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.160604 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj"] Nov 03 22:44:32.193930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.193895 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789a71e0-4x2gj\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:32.194229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.194205 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsg79\" (UniqueName: \"kubernetes.io/projected/2c964482-99ad-42ee-888e-f54a236a47e6-kube-api-access-xsg79\") pod \"maas-api-revoke-sub-789a71e0-4x2gj\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:32.194356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.194322 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789a71e0-4x2gj\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:32.295929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.295896 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsg79\" (UniqueName: \"kubernetes.io/projected/2c964482-99ad-42ee-888e-f54a236a47e6-kube-api-access-xsg79\") pod \"maas-api-revoke-sub-789a71e0-4x2gj\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:32.296131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.295977 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789a71e0-4x2gj\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:32.296131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.296070 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789a71e0-4x2gj\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:32.296677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.296639 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789a71e0-4x2gj\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:32.298540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.298520 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789a71e0-4x2gj\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:32.352943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.352882 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsg79\" (UniqueName: \"kubernetes.io/projected/2c964482-99ad-42ee-888e-f54a236a47e6-kube-api-access-xsg79\") pod \"maas-api-revoke-sub-789a71e0-4x2gj\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:32.418171 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.418144 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:32.718585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.718555 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:32.787711 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:44:32.787675 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c964482_99ad_42ee_888e_f54a236a47e6.slice/crio-ac906ffed963a1c6e822b3f7952f7bd4f0fa64d18a74c379b6e626e69d3491be WatchSource:0}: Error finding container ac906ffed963a1c6e822b3f7952f7bd4f0fa64d18a74c379b6e626e69d3491be: Status 404 returned error can't find the container with id ac906ffed963a1c6e822b3f7952f7bd4f0fa64d18a74c379b6e626e69d3491be Nov 03 22:44:32.792598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:32.792574 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj"] Nov 03 22:44:33.136482 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:33.136447 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c39859-96fe-412f-8cc9-63b7c5f29bea" path="/var/lib/kubelet/pods/52c39859-96fe-412f-8cc9-63b7c5f29bea/volumes" Nov 03 22:44:33.137017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:33.137000 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6549bc-c506-425a-bf96-9d3ba30b9d6c" path="/var/lib/kubelet/pods/5f6549bc-c506-425a-bf96-9d3ba30b9d6c/volumes" Nov 03 22:44:33.137434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:33.137421 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84461fb7-32f0-48ea-b8dd-f7a03d6b3191" path="/var/lib/kubelet/pods/84461fb7-32f0-48ea-b8dd-f7a03d6b3191/volumes" Nov 03 22:44:33.551302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:33.551264 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" event={"ID":"2c964482-99ad-42ee-888e-f54a236a47e6","Type":"ContainerStarted","Data":"690077cbadb6d0e756cbc3b8b0579d65b6f43e724c8267d703c12fa84e6a16ae"} Nov 03 22:44:33.551485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:33.551328 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" event={"ID":"2c964482-99ad-42ee-888e-f54a236a47e6","Type":"ContainerStarted","Data":"ac906ffed963a1c6e822b3f7952f7bd4f0fa64d18a74c379b6e626e69d3491be"} Nov 03 22:44:33.573603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:33.573565 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:33.673031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:33.672982 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" podStartSLOduration=1.6729660179999999 podStartE2EDuration="1.672966018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:33.668290891 +0000 UTC m=+2230.649024543" watchObservedRunningTime="2026-11-03 22:44:33.672966018 +0000 UTC m=+2230.653699671" Nov 03 22:44:33.778561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:33.778521 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:33.960264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:33.960229 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:34.478275 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:34.478222 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:34.561189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:34.561156 4298 generic.go:356] "Generic (PLEG): container finished" podID="2c964482-99ad-42ee-888e-f54a236a47e6" containerID="690077cbadb6d0e756cbc3b8b0579d65b6f43e724c8267d703c12fa84e6a16ae" exitCode=0 Nov 03 22:44:34.561361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:34.561239 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" event={"ID":"2c964482-99ad-42ee-888e-f54a236a47e6","Type":"ContainerDied","Data":"690077cbadb6d0e756cbc3b8b0579d65b6f43e724c8267d703c12fa84e6a16ae"} Nov 03 22:44:34.655188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:34.655149 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:34.736676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:34.736576 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll"] Nov 03 22:44:34.736886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:34.736859 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" containerID="cri-o://cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c" gracePeriod=30 Nov 03 22:44:34.843670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:34.838599 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:35.357011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:35.356982 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:35.718002 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:35.717977 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:35.848915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:35.848881 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c964482-99ad-42ee-888e-f54a236a47e6-kube-api-access-xsg79\" (UniqueName: \"kubernetes.io/projected/2c964482-99ad-42ee-888e-f54a236a47e6-kube-api-access-xsg79\") pod \"2c964482-99ad-42ee-888e-f54a236a47e6\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " Nov 03 22:44:35.849243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:35.849219 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-openshift-ca\") pod \"2c964482-99ad-42ee-888e-f54a236a47e6\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " Nov 03 22:44:35.849332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:35.849305 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-xks-ca\") pod \"2c964482-99ad-42ee-888e-f54a236a47e6\" (UID: \"2c964482-99ad-42ee-888e-f54a236a47e6\") " Nov 03 22:44:35.849677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:35.849623 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-openshift-ca" pod "2c964482-99ad-42ee-888e-f54a236a47e6" (UID: "2c964482-99ad-42ee-888e-f54a236a47e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:35.849977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:35.849951 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:35.851517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:35.851491 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-xks-ca" pod "2c964482-99ad-42ee-888e-f54a236a47e6" (UID: "2c964482-99ad-42ee-888e-f54a236a47e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:35.851637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:35.851613 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c964482-99ad-42ee-888e-f54a236a47e6-kube-api-access-xsg79" pod "2c964482-99ad-42ee-888e-f54a236a47e6" (UID: "2c964482-99ad-42ee-888e-f54a236a47e6"). InnerVolumeSpecName "kube-api-access-xsg79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:35.951315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:35.951289 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c964482-99ad-42ee-888e-f54a236a47e6-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:35.951475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:35.951317 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsg79\" (UniqueName: \"kubernetes.io/projected/2c964482-99ad-42ee-888e-f54a236a47e6-kube-api-access-xsg79\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:36.576523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:36.576493 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" Nov 03 22:44:36.576523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:36.576517 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj" event={"ID":"2c964482-99ad-42ee-888e-f54a236a47e6","Type":"ContainerDied","Data":"ac906ffed963a1c6e822b3f7952f7bd4f0fa64d18a74c379b6e626e69d3491be"} Nov 03 22:44:36.576776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:36.576558 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac906ffed963a1c6e822b3f7952f7bd4f0fa64d18a74c379b6e626e69d3491be" Nov 03 22:44:37.111514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.111477 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:37.224733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.220449 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj"] Nov 03 22:44:37.224733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.221430 4298 cpu_manager.go:414] "containerMap: removing container" podUID="2c964482-99ad-42ee-888e-f54a236a47e6" containerName="revoke-keys" Nov 03 22:44:37.224733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.221450 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c964482-99ad-42ee-888e-f54a236a47e6" containerName="revoke-keys" Nov 03 22:44:37.224733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.221658 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c964482-99ad-42ee-888e-f54a236a47e6" containerName="revoke-keys" Nov 03 22:44:37.254663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.247253 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:37.294761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.294722 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:37.303701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.303676 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:44:37.303701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.303693 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:44:37.362327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.362234 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj"] Nov 03 22:44:37.376608 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.376573 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:37.376844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.376632 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c67jq\" (UniqueName: \"kubernetes.io/projected/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-kube-api-access-c67jq\") pod \"maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:37.376844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.376808 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:37.478551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.478505 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:37.479040 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.478571 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c67jq\" (UniqueName: \"kubernetes.io/projected/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-kube-api-access-c67jq\") pod \"maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:37.479040 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.478734 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:37.479424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.479394 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:37.481577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.481553 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:37.488320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.488296 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq"] Nov 03 22:44:37.492753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.492734 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:44:37.536621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.536592 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c67jq\" (UniqueName: \"kubernetes.io/projected/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-kube-api-access-c67jq\") pod \"maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:37.546786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.546759 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-01bb3a-dockercfg-886hq\"" Nov 03 22:44:37.546786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.546807 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-01bb3a\"" Nov 03 22:44:37.549231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.549210 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq"] Nov 03 22:44:37.574125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.574088 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:37.581908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.580348 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e83e8da-abd2-4045-954c-d4e80b249aa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq\" (UID: \"9e83e8da-abd2-4045-954c-d4e80b249aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:44:37.581908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.580435 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm5kz\" (UniqueName: \"kubernetes.io/projected/9e83e8da-abd2-4045-954c-d4e80b249aa8-kube-api-access-nm5kz\") pod \"payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq\" (UID: \"9e83e8da-abd2-4045-954c-d4e80b249aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:44:37.644985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.644955 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs"] Nov 03 22:44:37.650289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.650264 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:44:37.682333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.682151 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:37.682333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.682213 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e83e8da-abd2-4045-954c-d4e80b249aa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq\" (UID: \"9e83e8da-abd2-4045-954c-d4e80b249aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:44:37.682656 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.682340 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm5kz\" (UniqueName: \"kubernetes.io/projected/9e83e8da-abd2-4045-954c-d4e80b249aa8-kube-api-access-nm5kz\") pod \"payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq\" (UID: \"9e83e8da-abd2-4045-954c-d4e80b249aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:44:37.684745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.683556 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e83e8da-abd2-4045-954c-d4e80b249aa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq\" (UID: \"9e83e8da-abd2-4045-954c-d4e80b249aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:44:37.793088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.793050 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm5kz\" (UniqueName: \"kubernetes.io/projected/9e83e8da-abd2-4045-954c-d4e80b249aa8-kube-api-access-nm5kz\") pod \"payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq\" (UID: \"9e83e8da-abd2-4045-954c-d4e80b249aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:44:37.799802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.799691 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs"] Nov 03 22:44:37.815997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.815970 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:44:37.889317 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.889282 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpdxt\" (UniqueName: \"kubernetes.io/projected/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-kube-api-access-bpdxt\") pod \"payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs\" (UID: \"2b394ca3-e524-45da-a176-5e3b1d4d4cf6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:44:37.889490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.889438 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs\" (UID: \"2b394ca3-e524-45da-a176-5e3b1d4d4cf6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:44:37.990546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.990501 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpdxt\" (UniqueName: \"kubernetes.io/projected/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-kube-api-access-bpdxt\") pod \"payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs\" (UID: \"2b394ca3-e524-45da-a176-5e3b1d4d4cf6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:44:37.990792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.990611 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs\" (UID: \"2b394ca3-e524-45da-a176-5e3b1d4d4cf6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:44:37.991380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:37.991356 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs\" (UID: \"2b394ca3-e524-45da-a176-5e3b1d4d4cf6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:44:37.996501 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:44:37.996473 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a7c0e8_0477_4a3e_9b63_dbd93fb2b556.slice/crio-aa30fcac9e9d3713e5c9438df3f33d4ef1aa25b614dad77d44b69d764a6597ec WatchSource:0}: Error finding container aa30fcac9e9d3713e5c9438df3f33d4ef1aa25b614dad77d44b69d764a6597ec: Status 404 returned error can't find the container with id aa30fcac9e9d3713e5c9438df3f33d4ef1aa25b614dad77d44b69d764a6597ec Nov 03 22:44:38.000684 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.000638 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj"] Nov 03 22:44:38.047962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.047930 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpdxt\" (UniqueName: \"kubernetes.io/projected/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-kube-api-access-bpdxt\") pod \"payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs\" (UID: \"2b394ca3-e524-45da-a176-5e3b1d4d4cf6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:44:38.198421 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:44:38.198387 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e83e8da_abd2_4045_954c_d4e80b249aa8.slice/crio-34c8c5f5bbabdf1759380c5e386827d4ce2fa1ee72e901d0fd08ce03ab67eb2d WatchSource:0}: Error finding container 34c8c5f5bbabdf1759380c5e386827d4ce2fa1ee72e901d0fd08ce03ab67eb2d: Status 404 returned error can't find the container with id 34c8c5f5bbabdf1759380c5e386827d4ce2fa1ee72e901d0fd08ce03ab67eb2d Nov 03 22:44:38.203088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.203065 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq"] Nov 03 22:44:38.265193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.265174 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:44:38.606092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.606002 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" event={"ID":"9e83e8da-abd2-4045-954c-d4e80b249aa8","Type":"ContainerStarted","Data":"b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47"} Nov 03 22:44:38.606092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.606061 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" event={"ID":"9e83e8da-abd2-4045-954c-d4e80b249aa8","Type":"ContainerStarted","Data":"34c8c5f5bbabdf1759380c5e386827d4ce2fa1ee72e901d0fd08ce03ab67eb2d"} Nov 03 22:44:38.606092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.606082 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:44:38.607720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.607692 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" event={"ID":"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556","Type":"ContainerStarted","Data":"484b99b91e6f3cd8a244885e6a03e68a07950f323f33a5b16e037d17cc8da45f"} Nov 03 22:44:38.607834 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.607743 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" event={"ID":"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556","Type":"ContainerStarted","Data":"aa30fcac9e9d3713e5c9438df3f33d4ef1aa25b614dad77d44b69d764a6597ec"} Nov 03 22:44:38.633230 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:44:38.633195 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b394ca3_e524_45da_a176_5e3b1d4d4cf6.slice/crio-72f2b7dd17045f24dd1d2c28aa291127b489ec919d7add57e16f4bac9d4616d1 WatchSource:0}: Error finding container 72f2b7dd17045f24dd1d2c28aa291127b489ec919d7add57e16f4bac9d4616d1: Status 404 returned error can't find the container with id 72f2b7dd17045f24dd1d2c28aa291127b489ec919d7add57e16f4bac9d4616d1 Nov 03 22:44:38.638045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.638022 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs"] Nov 03 22:44:38.721144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.721091 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" podStartSLOduration=1.7210732050000002 podStartE2EDuration="1.721073205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:38.719553354 +0000 UTC m=+2235.700287005" watchObservedRunningTime="2026-11-03 22:44:38.721073205 +0000 UTC m=+2235.701806859" Nov 03 22:44:38.851757 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:38.851688 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" podStartSLOduration=1.851620705 podStartE2EDuration="1.851620705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:38.833246429 +0000 UTC m=+2235.813980119" watchObservedRunningTime="2026-11-03 22:44:38.851620705 +0000 UTC m=+2235.832354361" Nov 03 22:44:39.618217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:39.618179 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" event={"ID":"2b394ca3-e524-45da-a176-5e3b1d4d4cf6","Type":"ContainerStarted","Data":"1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b"} Nov 03 22:44:39.618689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:39.618236 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" event={"ID":"2b394ca3-e524-45da-a176-5e3b1d4d4cf6","Type":"ContainerStarted","Data":"72f2b7dd17045f24dd1d2c28aa291127b489ec919d7add57e16f4bac9d4616d1"} Nov 03 22:44:39.618689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:39.618339 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:44:39.619968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:39.619944 4298 generic.go:356] "Generic (PLEG): container finished" podID="14a7c0e8-0477-4a3e-9b63-dbd93fb2b556" containerID="484b99b91e6f3cd8a244885e6a03e68a07950f323f33a5b16e037d17cc8da45f" exitCode=0 Nov 03 22:44:39.620081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:39.620023 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" event={"ID":"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556","Type":"ContainerDied","Data":"484b99b91e6f3cd8a244885e6a03e68a07950f323f33a5b16e037d17cc8da45f"} Nov 03 22:44:39.896028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:39.895922 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" podStartSLOduration=2.895907237 podStartE2EDuration="2.895907237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:39.781396362 +0000 UTC m=+2236.762130016" watchObservedRunningTime="2026-11-03 22:44:39.895907237 +0000 UTC m=+2236.876640889" Nov 03 22:44:40.042374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.042348 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:44:40.119368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119336 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-certs\") pod \"d8131b38-abea-4933-91b4-e217facd68d8\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " Nov 03 22:44:40.119539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119424 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8131b38-abea-4933-91b4-e217facd68d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8131b38-abea-4933-91b4-e217facd68d8-istio-podinfo\") pod \"d8131b38-abea-4933-91b4-e217facd68d8\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " Nov 03 22:44:40.119539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119448 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8131b38-abea-4933-91b4-e217facd68d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8131b38-abea-4933-91b4-e217facd68d8-istiod-ca-cert\") pod \"d8131b38-abea-4933-91b4-e217facd68d8\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " Nov 03 22:44:40.119539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119504 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-socket\") pod \"d8131b38-abea-4933-91b4-e217facd68d8\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " Nov 03 22:44:40.119710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119598 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-kube-api-access-bmvll\" (UniqueName: \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-kube-api-access-bmvll\") pod \"d8131b38-abea-4933-91b4-e217facd68d8\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " Nov 03 22:44:40.119710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119604 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-certs" pod "d8131b38-abea-4933-91b4-e217facd68d8" (UID: "d8131b38-abea-4933-91b4-e217facd68d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:40.119710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119695 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-data\") pod \"d8131b38-abea-4933-91b4-e217facd68d8\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " Nov 03 22:44:40.119888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119799 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-envoy\") pod \"d8131b38-abea-4933-91b4-e217facd68d8\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " Nov 03 22:44:40.119888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119824 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-socket" pod "d8131b38-abea-4933-91b4-e217facd68d8" (UID: "d8131b38-abea-4933-91b4-e217facd68d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:40.120008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119894 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8131b38-abea-4933-91b4-e217facd68d8-istiod-ca-cert" pod "d8131b38-abea-4933-91b4-e217facd68d8" (UID: "d8131b38-abea-4933-91b4-e217facd68d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:40.120008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119897 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-credential-socket\") pod \"d8131b38-abea-4933-91b4-e217facd68d8\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " Nov 03 22:44:40.120008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119950 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-istio-token\" (UniqueName: \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-istio-token\") pod \"d8131b38-abea-4933-91b4-e217facd68d8\" (UID: \"d8131b38-abea-4933-91b4-e217facd68d8\") " Nov 03 22:44:40.120008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.119952 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-data" pod "d8131b38-abea-4933-91b4-e217facd68d8" (UID: "d8131b38-abea-4933-91b4-e217facd68d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:40.120228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.120082 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-credential-socket" pod "d8131b38-abea-4933-91b4-e217facd68d8" (UID: "d8131b38-abea-4933-91b4-e217facd68d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:40.120593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.120574 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:40.120593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.120593 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:40.120803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.120604 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:40.120803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.120613 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8131b38-abea-4933-91b4-e217facd68d8-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:40.120803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.120623 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:40.122014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.121986 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-kube-api-access-bmvll" pod "d8131b38-abea-4933-91b4-e217facd68d8" (UID: "d8131b38-abea-4933-91b4-e217facd68d8"). InnerVolumeSpecName "kube-api-access-bmvll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:40.122122 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.122024 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-istio-token" pod "d8131b38-abea-4933-91b4-e217facd68d8" (UID: "d8131b38-abea-4933-91b4-e217facd68d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:40.122181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.122121 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-envoy" pod "d8131b38-abea-4933-91b4-e217facd68d8" (UID: "d8131b38-abea-4933-91b4-e217facd68d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:40.122226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.122184 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8131b38-abea-4933-91b4-e217facd68d8-istio-podinfo" pod "d8131b38-abea-4933-91b4-e217facd68d8" (UID: "d8131b38-abea-4933-91b4-e217facd68d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:40.222287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.222244 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmvll\" (UniqueName: \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-kube-api-access-bmvll\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:40.222556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.222286 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8131b38-abea-4933-91b4-e217facd68d8-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:40.222556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.222312 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8131b38-abea-4933-91b4-e217facd68d8-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:40.222556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.222325 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8131b38-abea-4933-91b4-e217facd68d8-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:40.281630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.281588 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl"] Nov 03 22:44:40.310375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.310345 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s"] Nov 03 22:44:40.481110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.480993 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:40.629654 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.629608 4298 generic.go:356] "Generic (PLEG): container finished" podID="d8131b38-abea-4933-91b4-e217facd68d8" containerID="cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c" exitCode=0 Nov 03 22:44:40.630102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.629673 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" Nov 03 22:44:40.630102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.629669 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" event={"ID":"d8131b38-abea-4933-91b4-e217facd68d8","Type":"ContainerDied","Data":"cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c"} Nov 03 22:44:40.630102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.629789 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll" event={"ID":"d8131b38-abea-4933-91b4-e217facd68d8","Type":"ContainerDied","Data":"d03c2b473f205eb24c47767e59546b0a62e8d98be76eb434c3a16194a9b66aa8"} Nov 03 22:44:40.630102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.629811 4298 scope.go:122] "RemoveContainer" containerID="cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c" Nov 03 22:44:40.642168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.642145 4298 scope.go:122] "RemoveContainer" containerID="4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74" Nov 03 22:44:40.652219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.652198 4298 scope.go:122] "RemoveContainer" containerID="cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c" Nov 03 22:44:40.652569 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:44:40.652540 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c\": container with ID starting with cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c not found: ID does not exist" containerID="cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c" Nov 03 22:44:40.652638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.652578 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c"} err="failed to get container status \"cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c\": rpc error: code = NotFound desc = could not find container \"cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c\": container with ID starting with cd03454cd0a4c5f7283893f8bf54f56b40f6485428ed409f6450489289c32a5c not found: ID does not exist" Nov 03 22:44:40.652638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.652596 4298 scope.go:122] "RemoveContainer" containerID="4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74" Nov 03 22:44:40.652860 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:44:40.652845 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74\": container with ID starting with 4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74 not found: ID does not exist" containerID="4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74" Nov 03 22:44:40.652907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.652863 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74"} err="failed to get container status \"4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74\": rpc error: code = NotFound desc = could not find container \"4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74\": container with ID starting with 4bae2ef963d529d52431613a134a98b392bed1babb018ecfecf0262e4a11be74 not found: ID does not exist" Nov 03 22:44:40.797922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.797892 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:40.798519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.798499 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll"] Nov 03 22:44:40.802980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.802958 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a9ef23-openshift-default-75db87b5bc-4rjll"] Nov 03 22:44:40.830291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.830267 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-kube-api-access-c67jq\" (UniqueName: \"kubernetes.io/projected/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-kube-api-access-c67jq\") pod \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " Nov 03 22:44:40.830430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.830312 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-xks-ca\") pod \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " Nov 03 22:44:40.830588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.830572 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-openshift-ca\") pod \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\" (UID: \"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556\") " Nov 03 22:44:40.830983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.830957 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-openshift-ca" pod "14a7c0e8-0477-4a3e-9b63-dbd93fb2b556" (UID: "14a7c0e8-0477-4a3e-9b63-dbd93fb2b556"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:40.831134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.831115 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:40.832467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.832442 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-xks-ca" pod "14a7c0e8-0477-4a3e-9b63-dbd93fb2b556" (UID: "14a7c0e8-0477-4a3e-9b63-dbd93fb2b556"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:40.832548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.832516 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-kube-api-access-c67jq" pod "14a7c0e8-0477-4a3e-9b63-dbd93fb2b556" (UID: "14a7c0e8-0477-4a3e-9b63-dbd93fb2b556"). InnerVolumeSpecName "kube-api-access-c67jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:40.932143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.932110 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c67jq\" (UniqueName: \"kubernetes.io/projected/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-kube-api-access-c67jq\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:40.932143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:40.932138 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:41.137155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:41.137072 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8131b38-abea-4933-91b4-e217facd68d8" path="/var/lib/kubelet/pods/d8131b38-abea-4933-91b4-e217facd68d8/volumes" Nov 03 22:44:41.639406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:41.639369 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" Nov 03 22:44:41.639406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:41.639385 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj" event={"ID":"14a7c0e8-0477-4a3e-9b63-dbd93fb2b556","Type":"ContainerDied","Data":"aa30fcac9e9d3713e5c9438df3f33d4ef1aa25b614dad77d44b69d764a6597ec"} Nov 03 22:44:41.639908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:41.639435 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa30fcac9e9d3713e5c9438df3f33d4ef1aa25b614dad77d44b69d764a6597ec" Nov 03 22:44:42.569887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:42.569828 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:42.833022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:42.832940 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:43.258897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.258866 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk"] Nov 03 22:44:43.259514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.259500 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" Nov 03 22:44:43.259575 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.259516 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" Nov 03 22:44:43.259575 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.259528 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" Nov 03 22:44:43.259575 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.259534 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" Nov 03 22:44:43.259575 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.259572 4298 cpu_manager.go:414] "containerMap: removing container" podUID="14a7c0e8-0477-4a3e-9b63-dbd93fb2b556" containerName="revoke-keys" Nov 03 22:44:43.259729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.259577 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a7c0e8-0477-4a3e-9b63-dbd93fb2b556" containerName="revoke-keys" Nov 03 22:44:43.259729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.259704 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a7c0e8-0477-4a3e-9b63-dbd93fb2b556" containerName="revoke-keys" Nov 03 22:44:43.259729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.259722 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" Nov 03 22:44:43.259729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.259729 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8131b38-abea-4933-91b4-e217facd68d8" containerName="istio-proxy" Nov 03 22:44:43.263720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.263702 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:43.316230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.316198 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:44:43.319707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.319678 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk"] Nov 03 22:44:43.361874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.361840 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:43.362127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.362103 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:43.362194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.362183 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bft26\" (UniqueName: \"kubernetes.io/projected/7c51ea65-3567-4f09-a0be-1187c75d33ac-kube-api-access-bft26\") pod \"maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:43.364579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.364556 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:44:43.463410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.463369 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:43.463629 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.463464 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bft26\" (UniqueName: \"kubernetes.io/projected/7c51ea65-3567-4f09-a0be-1187c75d33ac-kube-api-access-bft26\") pod \"maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:43.463629 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.463568 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:43.464086 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.464066 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:43.465985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.465958 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:43.520450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.520364 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bft26\" (UniqueName: \"kubernetes.io/projected/7c51ea65-3567-4f09-a0be-1187c75d33ac-kube-api-access-bft26\") pod \"maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:43.577473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.577437 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:43.947458 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:44:43.947423 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c51ea65_3567_4f09_a0be_1187c75d33ac.slice/crio-70411defd0eaf89e40217e810bd75a243171859c3641564c267d772392d5f944 WatchSource:0}: Error finding container 70411defd0eaf89e40217e810bd75a243171859c3641564c267d772392d5f944: Status 404 returned error can't find the container with id 70411defd0eaf89e40217e810bd75a243171859c3641564c267d772392d5f944 Nov 03 22:44:43.951903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:43.951880 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk"] Nov 03 22:44:44.667785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:44.667690 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" event={"ID":"7c51ea65-3567-4f09-a0be-1187c75d33ac","Type":"ContainerStarted","Data":"933548b571dc3d066e34c3541bce57c66b333155d43a01a518ac36093a92cc4a"} Nov 03 22:44:44.667785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:44.667750 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" event={"ID":"7c51ea65-3567-4f09-a0be-1187c75d33ac","Type":"ContainerStarted","Data":"70411defd0eaf89e40217e810bd75a243171859c3641564c267d772392d5f944"} Nov 03 22:44:44.783409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:44.783358 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" podStartSLOduration=1.783343204 podStartE2EDuration="1.783343204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:44.780424521 +0000 UTC m=+2241.761158176" watchObservedRunningTime="2026-11-03 22:44:44.783343204 +0000 UTC m=+2241.764076857" Nov 03 22:44:45.677163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:45.677127 4298 generic.go:356] "Generic (PLEG): container finished" podID="7c51ea65-3567-4f09-a0be-1187c75d33ac" containerID="933548b571dc3d066e34c3541bce57c66b333155d43a01a518ac36093a92cc4a" exitCode=0 Nov 03 22:44:45.677579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:45.677211 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" event={"ID":"7c51ea65-3567-4f09-a0be-1187c75d33ac","Type":"ContainerDied","Data":"933548b571dc3d066e34c3541bce57c66b333155d43a01a518ac36093a92cc4a"} Nov 03 22:44:46.834967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:46.834625 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:46.911232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:46.911197 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-xks-ca\") pod \"7c51ea65-3567-4f09-a0be-1187c75d33ac\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " Nov 03 22:44:46.911429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:46.911264 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-openshift-ca\") pod \"7c51ea65-3567-4f09-a0be-1187c75d33ac\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " Nov 03 22:44:46.911429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:46.911367 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c51ea65-3567-4f09-a0be-1187c75d33ac-kube-api-access-bft26\" (UniqueName: \"kubernetes.io/projected/7c51ea65-3567-4f09-a0be-1187c75d33ac-kube-api-access-bft26\") pod \"7c51ea65-3567-4f09-a0be-1187c75d33ac\" (UID: \"7c51ea65-3567-4f09-a0be-1187c75d33ac\") " Nov 03 22:44:46.911714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:46.911636 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-openshift-ca" pod "7c51ea65-3567-4f09-a0be-1187c75d33ac" (UID: "7c51ea65-3567-4f09-a0be-1187c75d33ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:46.912007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:46.911991 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:46.913534 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:46.913500 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c51ea65-3567-4f09-a0be-1187c75d33ac-kube-api-access-bft26" pod "7c51ea65-3567-4f09-a0be-1187c75d33ac" (UID: "7c51ea65-3567-4f09-a0be-1187c75d33ac"). InnerVolumeSpecName "kube-api-access-bft26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:46.913626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:46.913554 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-xks-ca" pod "7c51ea65-3567-4f09-a0be-1187c75d33ac" (UID: "7c51ea65-3567-4f09-a0be-1187c75d33ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:47.013282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:47.013249 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c51ea65-3567-4f09-a0be-1187c75d33ac-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:47.013282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:47.013279 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bft26\" (UniqueName: \"kubernetes.io/projected/7c51ea65-3567-4f09-a0be-1187c75d33ac-kube-api-access-bft26\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:47.493138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:47.493103 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:47.559479 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:44:47.559443 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:44:47.692567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:47.692538 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" Nov 03 22:44:47.692741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:47.692570 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk" event={"ID":"7c51ea65-3567-4f09-a0be-1187c75d33ac","Type":"ContainerDied","Data":"70411defd0eaf89e40217e810bd75a243171859c3641564c267d772392d5f944"} Nov 03 22:44:47.692741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:47.692621 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70411defd0eaf89e40217e810bd75a243171859c3641564c267d772392d5f944" Nov 03 22:44:47.802266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:47.802187 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x"] Nov 03 22:44:47.923442 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:47.923409 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:48.722975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:48.722947 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5"] Nov 03 22:44:49.127528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:49.127311 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:49.306686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:49.306624 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:49.623965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:49.623940 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:44:50.634078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:50.634045 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:44:52.490200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:52.490165 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:53.723300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:53.723265 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" podUID="62a08a87-7277-4fe5-817d-90a33f64e8cb" containerName="payload-processing" containerID="cri-o://9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00" gracePeriod=25 Nov 03 22:44:53.993589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:53.993527 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:44:54.125930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.125897 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a08a87-7277-4fe5-817d-90a33f64e8cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a08a87-7277-4fe5-817d-90a33f64e8cb-plugins-config-volume\") pod \"62a08a87-7277-4fe5-817d-90a33f64e8cb\" (UID: \"62a08a87-7277-4fe5-817d-90a33f64e8cb\") " Nov 03 22:44:54.126117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.126054 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a08a87-7277-4fe5-817d-90a33f64e8cb-kube-api-access-mztwv\" (UniqueName: \"kubernetes.io/projected/62a08a87-7277-4fe5-817d-90a33f64e8cb-kube-api-access-mztwv\") pod \"62a08a87-7277-4fe5-817d-90a33f64e8cb\" (UID: \"62a08a87-7277-4fe5-817d-90a33f64e8cb\") " Nov 03 22:44:54.126372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.126342 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a08a87-7277-4fe5-817d-90a33f64e8cb-plugins-config-volume" pod "62a08a87-7277-4fe5-817d-90a33f64e8cb" (UID: "62a08a87-7277-4fe5-817d-90a33f64e8cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:54.126804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.126778 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a08a87-7277-4fe5-817d-90a33f64e8cb-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:54.129314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.129283 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a08a87-7277-4fe5-817d-90a33f64e8cb-kube-api-access-mztwv" pod "62a08a87-7277-4fe5-817d-90a33f64e8cb" (UID: "62a08a87-7277-4fe5-817d-90a33f64e8cb"). InnerVolumeSpecName "kube-api-access-mztwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:54.228134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.228103 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mztwv\" (UniqueName: \"kubernetes.io/projected/62a08a87-7277-4fe5-817d-90a33f64e8cb-kube-api-access-mztwv\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:54.791802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.791768 4298 generic.go:356] "Generic (PLEG): container finished" podID="62a08a87-7277-4fe5-817d-90a33f64e8cb" containerID="9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00" exitCode=0 Nov 03 22:44:54.792257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.791825 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" Nov 03 22:44:54.792257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.791855 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" event={"ID":"62a08a87-7277-4fe5-817d-90a33f64e8cb","Type":"ContainerDied","Data":"9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00"} Nov 03 22:44:54.792257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.791907 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5" event={"ID":"62a08a87-7277-4fe5-817d-90a33f64e8cb","Type":"ContainerDied","Data":"334b928e9b51d8093ac64111a2fc008d46a319fbc642eee47c34a733465cfb77"} Nov 03 22:44:54.792257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.791933 4298 scope.go:122] "RemoveContainer" containerID="9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00" Nov 03 22:44:54.792802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.792770 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:54.802553 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.802538 4298 scope.go:122] "RemoveContainer" containerID="9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00" Nov 03 22:44:54.802850 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:44:54.802828 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00\": container with ID starting with 9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00 not found: ID does not exist" containerID="9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00" Nov 03 22:44:54.802899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.802857 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00"} err="failed to get container status \"9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00\": rpc error: code = NotFound desc = could not find container \"9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00\": container with ID starting with 9f8150360e6f5a2fea2c3c8883a7a2988e01026e4f402077cd7eef5f178fce00 not found: ID does not exist" Nov 03 22:44:54.986522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.986489 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5"] Nov 03 22:44:54.989120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.989086 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:54.994350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:54.994328 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a6b7f-856d764cc4-69rq5"] Nov 03 22:44:55.155337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.155262 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a08a87-7277-4fe5-817d-90a33f64e8cb" path="/var/lib/kubelet/pods/62a08a87-7277-4fe5-817d-90a33f64e8cb/volumes" Nov 03 22:44:55.304243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.304209 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" containerID="cri-o://ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e" gracePeriod=45 Nov 03 22:44:55.350116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.349929 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" containerID="cri-o://1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d" gracePeriod=45 Nov 03 22:44:55.390570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.390541 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused" Nov 03 22:44:55.666990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.666962 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:44:55.680477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.680451 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-home\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-home\") pod \"0f816022-7875-4505-99ff-072c84a5dc79\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " Nov 03 22:44:55.680618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.680520 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f816022-7875-4505-99ff-072c84a5dc79-kube-api-access-8c228\" (UniqueName: \"kubernetes.io/projected/0f816022-7875-4505-99ff-072c84a5dc79-kube-api-access-8c228\") pod \"0f816022-7875-4505-99ff-072c84a5dc79\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " Nov 03 22:44:55.680618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.680579 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f816022-7875-4505-99ff-072c84a5dc79-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f816022-7875-4505-99ff-072c84a5dc79-tls-certs\") pod \"0f816022-7875-4505-99ff-072c84a5dc79\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " Nov 03 22:44:55.680618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.680614 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-tmp-dir\") pod \"0f816022-7875-4505-99ff-072c84a5dc79\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " Nov 03 22:44:55.680816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.680727 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-dshm\") pod \"0f816022-7875-4505-99ff-072c84a5dc79\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " Nov 03 22:44:55.680816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.680749 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-model-cache\") pod \"0f816022-7875-4505-99ff-072c84a5dc79\" (UID: \"0f816022-7875-4505-99ff-072c84a5dc79\") " Nov 03 22:44:55.681579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.681351 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-home" pod "0f816022-7875-4505-99ff-072c84a5dc79" (UID: "0f816022-7875-4505-99ff-072c84a5dc79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:55.681787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.681700 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-tmp-dir" pod "0f816022-7875-4505-99ff-072c84a5dc79" (UID: "0f816022-7875-4505-99ff-072c84a5dc79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:55.682017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.681972 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-model-cache" pod "0f816022-7875-4505-99ff-072c84a5dc79" (UID: "0f816022-7875-4505-99ff-072c84a5dc79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:55.683797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.683768 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-dshm" pod "0f816022-7875-4505-99ff-072c84a5dc79" (UID: "0f816022-7875-4505-99ff-072c84a5dc79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:55.684049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.683963 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f816022-7875-4505-99ff-072c84a5dc79-tls-certs" pod "0f816022-7875-4505-99ff-072c84a5dc79" (UID: "0f816022-7875-4505-99ff-072c84a5dc79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:55.684049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.683983 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f816022-7875-4505-99ff-072c84a5dc79-kube-api-access-8c228" pod "0f816022-7875-4505-99ff-072c84a5dc79" (UID: "0f816022-7875-4505-99ff-072c84a5dc79"). InnerVolumeSpecName "kube-api-access-8c228". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:55.696015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.695994 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:44:55.781758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.781616 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-tmp-dir\") pod \"fee74c97-aedc-4db7-a744-c3a7af678d05\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " Nov 03 22:44:55.781758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.781751 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee74c97-aedc-4db7-a744-c3a7af678d05-kube-api-access-dldn9\" (UniqueName: \"kubernetes.io/projected/fee74c97-aedc-4db7-a744-c3a7af678d05-kube-api-access-dldn9\") pod \"fee74c97-aedc-4db7-a744-c3a7af678d05\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " Nov 03 22:44:55.782097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.781823 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-dshm\") pod \"fee74c97-aedc-4db7-a744-c3a7af678d05\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " Nov 03 22:44:55.782097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.781957 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-tmp-dir" pod "fee74c97-aedc-4db7-a744-c3a7af678d05" (UID: "fee74c97-aedc-4db7-a744-c3a7af678d05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:55.782097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782005 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee74c97-aedc-4db7-a744-c3a7af678d05-tls-certs\" (UniqueName: \"kubernetes.io/secret/fee74c97-aedc-4db7-a744-c3a7af678d05-tls-certs\") pod \"fee74c97-aedc-4db7-a744-c3a7af678d05\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " Nov 03 22:44:55.782277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782105 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-model-cache\") pod \"fee74c97-aedc-4db7-a744-c3a7af678d05\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " Nov 03 22:44:55.782277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782151 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-home\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-home\") pod \"fee74c97-aedc-4db7-a744-c3a7af678d05\" (UID: \"fee74c97-aedc-4db7-a744-c3a7af678d05\") " Nov 03 22:44:55.782442 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782414 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-model-cache" pod "fee74c97-aedc-4db7-a744-c3a7af678d05" (UID: "fee74c97-aedc-4db7-a744-c3a7af678d05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:55.782693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782669 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-home" pod "fee74c97-aedc-4db7-a744-c3a7af678d05" (UID: "fee74c97-aedc-4db7-a744-c3a7af678d05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:55.782826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782797 4298 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-home\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:55.782826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782820 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c228\" (UniqueName: \"kubernetes.io/projected/0f816022-7875-4505-99ff-072c84a5dc79-kube-api-access-8c228\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:55.782956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782836 4298 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f816022-7875-4505-99ff-072c84a5dc79-tls-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:55.782956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782851 4298 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-tmp-dir\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:55.782956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782864 4298 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-model-cache\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:55.782956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782880 4298 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-tmp-dir\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:55.782956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782893 4298 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-dshm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:55.782956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.782907 4298 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f816022-7875-4505-99ff-072c84a5dc79-model-cache\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:55.788180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.784229 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee74c97-aedc-4db7-a744-c3a7af678d05-kube-api-access-dldn9" pod "fee74c97-aedc-4db7-a744-c3a7af678d05" (UID: "fee74c97-aedc-4db7-a744-c3a7af678d05"). InnerVolumeSpecName "kube-api-access-dldn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:55.788180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.785404 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-dshm" pod "fee74c97-aedc-4db7-a744-c3a7af678d05" (UID: "fee74c97-aedc-4db7-a744-c3a7af678d05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:55.788180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.785537 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee74c97-aedc-4db7-a744-c3a7af678d05-tls-certs" pod "fee74c97-aedc-4db7-a744-c3a7af678d05" (UID: "fee74c97-aedc-4db7-a744-c3a7af678d05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:55.806586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.806530 4298 generic.go:356] "Generic (PLEG): container finished" podID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerID="1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d" exitCode=0 Nov 03 22:44:55.807089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.806803 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" event={"ID":"fee74c97-aedc-4db7-a744-c3a7af678d05","Type":"ContainerDied","Data":"1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d"} Nov 03 22:44:55.807089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.806840 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" event={"ID":"fee74c97-aedc-4db7-a744-c3a7af678d05","Type":"ContainerDied","Data":"9f2a5347da424d9814369d4114a2357e1888257b60acbd27934e4f691975d19d"} Nov 03 22:44:55.807089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.806861 4298 scope.go:122] "RemoveContainer" containerID="1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d" Nov 03 22:44:55.807089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.806957 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s" Nov 03 22:44:55.812549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.812454 4298 generic.go:356] "Generic (PLEG): container finished" podID="0f816022-7875-4505-99ff-072c84a5dc79" containerID="ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e" exitCode=0 Nov 03 22:44:55.812549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.812500 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" event={"ID":"0f816022-7875-4505-99ff-072c84a5dc79","Type":"ContainerDied","Data":"ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e"} Nov 03 22:44:55.812549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.812530 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" Nov 03 22:44:55.812793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.812533 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl" event={"ID":"0f816022-7875-4505-99ff-072c84a5dc79","Type":"ContainerDied","Data":"10b19c889fd5dd1ce871fd1f9d090a0b0c3de41a9b9226d228de45f8e416b261"} Nov 03 22:44:55.849900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.834856 4298 scope.go:122] "RemoveContainer" containerID="1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d" Nov 03 22:44:55.854995 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:44:55.854960 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d\": container with ID starting with 1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d not found: ID does not exist" containerID="1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d" Nov 03 22:44:55.855161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.855007 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d"} err="failed to get container status \"1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d\": rpc error: code = NotFound desc = could not find container \"1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d\": container with ID starting with 1ae35ca0573b9581d8c93e36be40f2b101714f51a7b847684b67fb6c93881b0d not found: ID does not exist" Nov 03 22:44:55.855161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.855032 4298 scope.go:122] "RemoveContainer" containerID="ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e" Nov 03 22:44:55.871296 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.871274 4298 scope.go:122] "RemoveContainer" containerID="ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e" Nov 03 22:44:55.871634 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:44:55.871613 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e\": container with ID starting with ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e not found: ID does not exist" containerID="ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e" Nov 03 22:44:55.871755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.871677 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e"} err="failed to get container status \"ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e\": rpc error: code = NotFound desc = could not find container \"ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e\": container with ID starting with ef7cf0c01ef76a16c25d0b29a5950fbb30d1ca9f95b8454cc1d39c95eeb0597e not found: ID does not exist" Nov 03 22:44:55.884056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.884030 4298 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-home\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:55.884156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.884063 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dldn9\" (UniqueName: \"kubernetes.io/projected/fee74c97-aedc-4db7-a744-c3a7af678d05-kube-api-access-dldn9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:55.884156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.884078 4298 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee74c97-aedc-4db7-a744-c3a7af678d05-dshm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:55.884156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:55.884092 4298 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee74c97-aedc-4db7-a744-c3a7af678d05-tls-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.123582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:56.123493 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s"] Nov 03 22:44:56.159351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:56.159296 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/simulator-w6-6a6b7f-kserve-5fccdf5f7c-r9w7s"] Nov 03 22:44:56.185457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:56.185425 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl"] Nov 03 22:44:56.191312 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:56.191280 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-2-w6-6a6b7f-kserve-5d669dbd76-t86kl"] Nov 03 22:44:57.161365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:57.161321 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f816022-7875-4505-99ff-072c84a5dc79" path="/var/lib/kubelet/pods/0f816022-7875-4505-99ff-072c84a5dc79/volumes" Nov 03 22:44:57.161972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:57.161949 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" path="/var/lib/kubelet/pods/fee74c97-aedc-4db7-a744-c3a7af678d05/volumes" Nov 03 22:44:57.786771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:44:57.786734 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:00.166598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.166567 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6"] Nov 03 22:45:00.167551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167530 4298 cpu_manager.go:414] "containerMap: removing container" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" Nov 03 22:45:00.167600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167555 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" Nov 03 22:45:00.167600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167585 4298 cpu_manager.go:414] "containerMap: removing container" podUID="7c51ea65-3567-4f09-a0be-1187c75d33ac" containerName="revoke-keys" Nov 03 22:45:00.167600 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167592 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c51ea65-3567-4f09-a0be-1187c75d33ac" containerName="revoke-keys" Nov 03 22:45:00.167722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167608 4298 cpu_manager.go:414] "containerMap: removing container" podUID="62a08a87-7277-4fe5-817d-90a33f64e8cb" containerName="payload-processing" Nov 03 22:45:00.167722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167616 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a08a87-7277-4fe5-817d-90a33f64e8cb" containerName="payload-processing" Nov 03 22:45:00.167722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167669 4298 cpu_manager.go:414] "containerMap: removing container" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" Nov 03 22:45:00.167722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167677 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" Nov 03 22:45:00.167849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167815 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f816022-7875-4505-99ff-072c84a5dc79" containerName="main" Nov 03 22:45:00.167849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167833 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee74c97-aedc-4db7-a744-c3a7af678d05" containerName="main" Nov 03 22:45:00.167849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167845 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c51ea65-3567-4f09-a0be-1187c75d33ac" containerName="revoke-keys" Nov 03 22:45:00.167942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.167854 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a08a87-7277-4fe5-817d-90a33f64e8cb" containerName="payload-processing" Nov 03 22:45:00.172006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.171986 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" Nov 03 22:45:00.172403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.172382 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz"] Nov 03 22:45:00.176701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.176677 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" Nov 03 22:45:00.177085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.177062 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn"] Nov 03 22:45:00.181256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.181239 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" Nov 03 22:45:00.216634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.216607 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6"] Nov 03 22:45:00.220819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.220802 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" Nov 03 22:45:00.220923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.220884 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp"] Nov 03 22:45:00.224922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.224902 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" Nov 03 22:45:00.226962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.226936 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b"] Nov 03 22:45:00.232617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.232597 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6"] Nov 03 22:45:00.232758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.232741 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" Nov 03 22:45:00.238269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.238247 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght"] Nov 03 22:45:00.238396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.238377 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" Nov 03 22:45:00.244162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.244140 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd"] Nov 03 22:45:00.244270 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.244237 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" Nov 03 22:45:00.248856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.248835 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr"] Nov 03 22:45:00.248972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.248937 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" Nov 03 22:45:00.253944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.253926 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" Nov 03 22:45:00.262536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.262509 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:45:00.270852 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.270832 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr"] Nov 03 22:45:00.287477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.287450 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn"] Nov 03 22:45:00.287880 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.287857 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" Nov 03 22:45:00.293574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.293541 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz"] Nov 03 22:45:00.301131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.301106 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6"] Nov 03 22:45:00.309195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309164 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3056f50-50c4-4804-af55-0e9e4f06f583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp\" (UID: \"f3056f50-50c4-4804-af55-0e9e4f06f583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" Nov 03 22:45:00.309319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309257 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22571f32-7943-4e26-96f5-f409490f2969-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b\" (UID: \"22571f32-7943-4e26-96f5-f409490f2969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" Nov 03 22:45:00.309319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309288 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2fk\" (UniqueName: \"kubernetes.io/projected/50000cc4-9bd6-4cfb-a85f-15b0ae191071-kube-api-access-ml2fk\") pod \"maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn\" (UID: \"50000cc4-9bd6-4cfb-a85f-15b0ae191071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" Nov 03 22:45:00.309447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309324 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5cl5\" (UniqueName: \"kubernetes.io/projected/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-kube-api-access-r5cl5\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz\" (UID: \"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" Nov 03 22:45:00.309447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309359 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whsdj\" (UniqueName: \"kubernetes.io/projected/1534c655-737d-499d-8fcf-00e85c96854c-kube-api-access-whsdj\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd\" (UID: \"1534c655-737d-499d-8fcf-00e85c96854c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" Nov 03 22:45:00.309447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309433 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtlms\" (UniqueName: \"kubernetes.io/projected/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-kube-api-access-jtlms\") pod \"maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6\" (UID: \"ecc2af8c-c3b6-4bde-ac6f-398533e20acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309472 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f863b3-9457-4888-b046-3cfd408e504a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr\" (UID: \"81f863b3-9457-4888-b046-3cfd408e504a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309500 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9948q\" (UniqueName: \"kubernetes.io/projected/03b37944-6808-42f4-857f-57ea8635a2cd-kube-api-access-9948q\") pod \"maas-api-key-cleanup-29895765-vbght\" (UID: \"03b37944-6808-42f4-857f-57ea8635a2cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309535 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjg2z\" (UniqueName: \"kubernetes.io/projected/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-kube-api-access-cjg2z\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6\" (UID: \"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309572 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1534c655-737d-499d-8fcf-00e85c96854c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd\" (UID: \"1534c655-737d-499d-8fcf-00e85c96854c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309596 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghxcc\" (UniqueName: \"kubernetes.io/projected/22571f32-7943-4e26-96f5-f409490f2969-kube-api-access-ghxcc\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b\" (UID: \"22571f32-7943-4e26-96f5-f409490f2969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309631 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qs8k\" (UniqueName: \"kubernetes.io/projected/c029cafa-94ab-4b6a-b775-3fee6c1acea6-kube-api-access-8qs8k\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6\" (UID: \"c029cafa-94ab-4b6a-b775-3fee6c1acea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309704 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzz9\" (UniqueName: \"kubernetes.io/projected/81f863b3-9457-4888-b046-3cfd408e504a-kube-api-access-9nzz9\") pod \"maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr\" (UID: \"81f863b3-9457-4888-b046-3cfd408e504a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309773 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c029cafa-94ab-4b6a-b775-3fee6c1acea6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6\" (UID: \"c029cafa-94ab-4b6a-b775-3fee6c1acea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309862 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b37944-6808-42f4-857f-57ea8635a2cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895765-vbght\" (UID: \"03b37944-6808-42f4-857f-57ea8635a2cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309896 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz\" (UID: \"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309919 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6\" (UID: \"ecc2af8c-c3b6-4bde-ac6f-398533e20acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.309981 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50000cc4-9bd6-4cfb-a85f-15b0ae191071-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn\" (UID: \"50000cc4-9bd6-4cfb-a85f-15b0ae191071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.310011 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkcz\" (UniqueName: \"kubernetes.io/projected/f3056f50-50c4-4804-af55-0e9e4f06f583-kube-api-access-cdkcz\") pod \"maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp\" (UID: \"f3056f50-50c4-4804-af55-0e9e4f06f583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" Nov 03 22:45:00.311483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.310039 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6\" (UID: \"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" Nov 03 22:45:00.324725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.324701 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp"] Nov 03 22:45:00.330768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.330742 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd"] Nov 03 22:45:00.375143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.375109 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr"] Nov 03 22:45:00.395915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.394330 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6"] Nov 03 22:45:00.396274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.396246 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr"] Nov 03 22:45:00.405175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.402613 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6"] Nov 03 22:45:00.409397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.409370 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b"] Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.413825 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50000cc4-9bd6-4cfb-a85f-15b0ae191071-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn\" (UID: \"50000cc4-9bd6-4cfb-a85f-15b0ae191071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.413869 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkcz\" (UniqueName: \"kubernetes.io/projected/f3056f50-50c4-4804-af55-0e9e4f06f583-kube-api-access-cdkcz\") pod \"maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp\" (UID: \"f3056f50-50c4-4804-af55-0e9e4f06f583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.413906 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6\" (UID: \"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.413946 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8jhx\" (UniqueName: \"kubernetes.io/projected/157ee087-43db-4f2f-879d-a6e477ddf041-kube-api-access-t8jhx\") pod \"maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr\" (UID: \"157ee087-43db-4f2f-879d-a6e477ddf041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.413979 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3056f50-50c4-4804-af55-0e9e4f06f583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp\" (UID: \"f3056f50-50c4-4804-af55-0e9e4f06f583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414061 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22571f32-7943-4e26-96f5-f409490f2969-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b\" (UID: \"22571f32-7943-4e26-96f5-f409490f2969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414090 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2fk\" (UniqueName: \"kubernetes.io/projected/50000cc4-9bd6-4cfb-a85f-15b0ae191071-kube-api-access-ml2fk\") pod \"maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn\" (UID: \"50000cc4-9bd6-4cfb-a85f-15b0ae191071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414128 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5cl5\" (UniqueName: \"kubernetes.io/projected/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-kube-api-access-r5cl5\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz\" (UID: \"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414157 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whsdj\" (UniqueName: \"kubernetes.io/projected/1534c655-737d-499d-8fcf-00e85c96854c-kube-api-access-whsdj\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd\" (UID: \"1534c655-737d-499d-8fcf-00e85c96854c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414227 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtlms\" (UniqueName: \"kubernetes.io/projected/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-kube-api-access-jtlms\") pod \"maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6\" (UID: \"ecc2af8c-c3b6-4bde-ac6f-398533e20acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414262 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f863b3-9457-4888-b046-3cfd408e504a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr\" (UID: \"81f863b3-9457-4888-b046-3cfd408e504a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414290 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9948q\" (UniqueName: \"kubernetes.io/projected/03b37944-6808-42f4-857f-57ea8635a2cd-kube-api-access-9948q\") pod \"maas-api-key-cleanup-29895765-vbght\" (UID: \"03b37944-6808-42f4-857f-57ea8635a2cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414326 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjg2z\" (UniqueName: \"kubernetes.io/projected/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-kube-api-access-cjg2z\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6\" (UID: \"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414363 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1534c655-737d-499d-8fcf-00e85c96854c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd\" (UID: \"1534c655-737d-499d-8fcf-00e85c96854c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414389 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghxcc\" (UniqueName: \"kubernetes.io/projected/22571f32-7943-4e26-96f5-f409490f2969-kube-api-access-ghxcc\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b\" (UID: \"22571f32-7943-4e26-96f5-f409490f2969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414427 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qs8k\" (UniqueName: \"kubernetes.io/projected/c029cafa-94ab-4b6a-b775-3fee6c1acea6-kube-api-access-8qs8k\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6\" (UID: \"c029cafa-94ab-4b6a-b775-3fee6c1acea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414478 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzz9\" (UniqueName: \"kubernetes.io/projected/81f863b3-9457-4888-b046-3cfd408e504a-kube-api-access-9nzz9\") pod \"maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr\" (UID: \"81f863b3-9457-4888-b046-3cfd408e504a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414543 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/157ee087-43db-4f2f-879d-a6e477ddf041-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr\" (UID: \"157ee087-43db-4f2f-879d-a6e477ddf041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414576 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c029cafa-94ab-4b6a-b775-3fee6c1acea6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6\" (UID: \"c029cafa-94ab-4b6a-b775-3fee6c1acea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414682 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b37944-6808-42f4-857f-57ea8635a2cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895765-vbght\" (UID: \"03b37944-6808-42f4-857f-57ea8635a2cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414716 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz\" (UID: \"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414745 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6\" (UID: \"ecc2af8c-c3b6-4bde-ac6f-398533e20acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.414873 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght"] Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.415520 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6\" (UID: \"ecc2af8c-c3b6-4bde-ac6f-398533e20acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.415630 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f863b3-9457-4888-b046-3cfd408e504a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr\" (UID: \"81f863b3-9457-4888-b046-3cfd408e504a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.416326 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50000cc4-9bd6-4cfb-a85f-15b0ae191071-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn\" (UID: \"50000cc4-9bd6-4cfb-a85f-15b0ae191071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.416583 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1534c655-737d-499d-8fcf-00e85c96854c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd\" (UID: \"1534c655-737d-499d-8fcf-00e85c96854c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" Nov 03 22:45:00.417549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.417175 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6\" (UID: \"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" Nov 03 22:45:00.419167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.417744 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c029cafa-94ab-4b6a-b775-3fee6c1acea6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6\" (UID: \"c029cafa-94ab-4b6a-b775-3fee6c1acea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" Nov 03 22:45:00.419167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.417819 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3056f50-50c4-4804-af55-0e9e4f06f583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp\" (UID: \"f3056f50-50c4-4804-af55-0e9e4f06f583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" Nov 03 22:45:00.419167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.418385 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b37944-6808-42f4-857f-57ea8635a2cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895765-vbght\" (UID: \"03b37944-6808-42f4-857f-57ea8635a2cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" Nov 03 22:45:00.419167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.418394 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22571f32-7943-4e26-96f5-f409490f2969-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b\" (UID: \"22571f32-7943-4e26-96f5-f409490f2969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" Nov 03 22:45:00.419390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.419354 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz\" (UID: \"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" Nov 03 22:45:00.474030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.473968 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjg2z\" (UniqueName: \"kubernetes.io/projected/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-kube-api-access-cjg2z\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6\" (UID: \"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" Nov 03 22:45:00.474030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.473988 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkcz\" (UniqueName: \"kubernetes.io/projected/f3056f50-50c4-4804-af55-0e9e4f06f583-kube-api-access-cdkcz\") pod \"maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp\" (UID: \"f3056f50-50c4-4804-af55-0e9e4f06f583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" Nov 03 22:45:00.474250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.474107 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qs8k\" (UniqueName: \"kubernetes.io/projected/c029cafa-94ab-4b6a-b775-3fee6c1acea6-kube-api-access-8qs8k\") pod \"maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6\" (UID: \"c029cafa-94ab-4b6a-b775-3fee6c1acea6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" Nov 03 22:45:00.474319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.474244 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9948q\" (UniqueName: \"kubernetes.io/projected/03b37944-6808-42f4-857f-57ea8635a2cd-kube-api-access-9948q\") pod \"maas-api-key-cleanup-29895765-vbght\" (UID: \"03b37944-6808-42f4-857f-57ea8635a2cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" Nov 03 22:45:00.474957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.474939 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghxcc\" (UniqueName: \"kubernetes.io/projected/22571f32-7943-4e26-96f5-f409490f2969-kube-api-access-ghxcc\") pod \"maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b\" (UID: \"22571f32-7943-4e26-96f5-f409490f2969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" Nov 03 22:45:00.475919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.475895 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzz9\" (UniqueName: \"kubernetes.io/projected/81f863b3-9457-4888-b046-3cfd408e504a-kube-api-access-9nzz9\") pod \"maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr\" (UID: \"81f863b3-9457-4888-b046-3cfd408e504a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" Nov 03 22:45:00.487351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.487322 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtlms\" (UniqueName: \"kubernetes.io/projected/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-kube-api-access-jtlms\") pod \"maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6\" (UID: \"ecc2af8c-c3b6-4bde-ac6f-398533e20acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" Nov 03 22:45:00.487796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.487772 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2fk\" (UniqueName: \"kubernetes.io/projected/50000cc4-9bd6-4cfb-a85f-15b0ae191071-kube-api-access-ml2fk\") pod \"maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn\" (UID: \"50000cc4-9bd6-4cfb-a85f-15b0ae191071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" Nov 03 22:45:00.487903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.487819 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5cl5\" (UniqueName: \"kubernetes.io/projected/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-kube-api-access-r5cl5\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz\" (UID: \"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" Nov 03 22:45:00.488336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.488319 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" Nov 03 22:45:00.488563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.488545 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whsdj\" (UniqueName: \"kubernetes.io/projected/1534c655-737d-499d-8fcf-00e85c96854c-kube-api-access-whsdj\") pod \"maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd\" (UID: \"1534c655-737d-499d-8fcf-00e85c96854c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" Nov 03 22:45:00.497088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.497070 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" Nov 03 22:45:00.507242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.502917 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" Nov 03 22:45:00.517314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.516727 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/157ee087-43db-4f2f-879d-a6e477ddf041-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr\" (UID: \"157ee087-43db-4f2f-879d-a6e477ddf041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" Nov 03 22:45:00.517314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.516884 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8jhx\" (UniqueName: \"kubernetes.io/projected/157ee087-43db-4f2f-879d-a6e477ddf041-kube-api-access-t8jhx\") pod \"maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr\" (UID: \"157ee087-43db-4f2f-879d-a6e477ddf041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" Nov 03 22:45:00.517483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.517386 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/157ee087-43db-4f2f-879d-a6e477ddf041-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr\" (UID: \"157ee087-43db-4f2f-879d-a6e477ddf041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" Nov 03 22:45:00.541461 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.540901 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" Nov 03 22:45:00.552666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.552614 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" Nov 03 22:45:00.566988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.564866 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" Nov 03 22:45:00.573148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.571976 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" Nov 03 22:45:00.582612 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.582246 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8jhx\" (UniqueName: \"kubernetes.io/projected/157ee087-43db-4f2f-879d-a6e477ddf041-kube-api-access-t8jhx\") pod \"maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr\" (UID: \"157ee087-43db-4f2f-879d-a6e477ddf041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" Nov 03 22:45:00.606233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.602812 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" Nov 03 22:45:00.606233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.606049 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" Nov 03 22:45:00.615855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.615121 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" Nov 03 22:45:00.630162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:00.627462 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" Nov 03 22:45:01.102485 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:45:01.102451 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc029cafa_94ab_4b6a_b775_3fee6c1acea6.slice/crio-d150b45413534c29dfe15b12fdaab333b0bec9086468d51ffdee4807d4f0a779 WatchSource:0}: Error finding container d150b45413534c29dfe15b12fdaab333b0bec9086468d51ffdee4807d4f0a779: Status 404 returned error can't find the container with id d150b45413534c29dfe15b12fdaab333b0bec9086468d51ffdee4807d4f0a779 Nov 03 22:45:01.105457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.105435 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6"] Nov 03 22:45:01.147138 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:45:01.147107 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c34f8d_7c95_4d4d_bfe9_0f5365ef0d12.slice/crio-81639ad591033d5bfabe59af3e4fa23236bf45bd6294c047761b9a2f04f72ad5 WatchSource:0}: Error finding container 81639ad591033d5bfabe59af3e4fa23236bf45bd6294c047761b9a2f04f72ad5: Status 404 returned error can't find the container with id 81639ad591033d5bfabe59af3e4fa23236bf45bd6294c047761b9a2f04f72ad5 Nov 03 22:45:01.147816 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:45:01.147752 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c WatchSource:0}: Error finding container c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c: Status 404 returned error can't find the container with id c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c Nov 03 22:45:01.153690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.153670 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz"] Nov 03 22:45:01.159381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.159356 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn"] Nov 03 22:45:01.193696 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:45:01.193668 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22571f32_7943_4e26_96f5_f409490f2969.slice/crio-8189f375f9d5a3a906d3df3b68d90b75615bb3321c10bcd0264ed9881ec690c7 WatchSource:0}: Error finding container 8189f375f9d5a3a906d3df3b68d90b75615bb3321c10bcd0264ed9881ec690c7: Status 404 returned error can't find the container with id 8189f375f9d5a3a906d3df3b68d90b75615bb3321c10bcd0264ed9881ec690c7 Nov 03 22:45:01.199015 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.198916 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b"] Nov 03 22:45:01.240003 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:45:01.239663 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b37944_6808_42f4_857f_57ea8635a2cd.slice/crio-c785d0aa0ff0c3d6466d164c83fd405c954295e858f561dc337e55658650c6cd WatchSource:0}: Error finding container c785d0aa0ff0c3d6466d164c83fd405c954295e858f561dc337e55658650c6cd: Status 404 returned error can't find the container with id c785d0aa0ff0c3d6466d164c83fd405c954295e858f561dc337e55658650c6cd Nov 03 22:45:01.244039 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:45:01.244008 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3056f50_50c4_4804_af55_0e9e4f06f583.slice/crio-d7ca4bf48b3b757e975d4712a0b454844d982152995c516866a32136c1acba03 WatchSource:0}: Error finding container d7ca4bf48b3b757e975d4712a0b454844d982152995c516866a32136c1acba03: Status 404 returned error can't find the container with id d7ca4bf48b3b757e975d4712a0b454844d982152995c516866a32136c1acba03 Nov 03 22:45:01.245134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.245114 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght"] Nov 03 22:45:01.249994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.249928 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp"] Nov 03 22:45:01.279595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.279408 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6"] Nov 03 22:45:01.286294 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.286267 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6"] Nov 03 22:45:01.328055 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:45:01.328022 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc2af8c_c3b6_4bde_ac6f_398533e20acf.slice/crio-599451bdfedd4f713b55ff875811fa056222455f853ef706ab476fdf46fe16f4 WatchSource:0}: Error finding container 599451bdfedd4f713b55ff875811fa056222455f853ef706ab476fdf46fe16f4: Status 404 returned error can't find the container with id 599451bdfedd4f713b55ff875811fa056222455f853ef706ab476fdf46fe16f4 Nov 03 22:45:01.341708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.341684 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd"] Nov 03 22:45:01.361865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.361840 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr"] Nov 03 22:45:01.384256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.384233 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr"] Nov 03 22:45:01.534361 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:45:01.534334 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef WatchSource:0}: Error finding container d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef: Status 404 returned error can't find the container with id d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef Nov 03 22:45:01.900070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.900017 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" event={"ID":"ecc2af8c-c3b6-4bde-ac6f-398533e20acf","Type":"ContainerStarted","Data":"599451bdfedd4f713b55ff875811fa056222455f853ef706ab476fdf46fe16f4"} Nov 03 22:45:01.901798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.901723 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" event={"ID":"03b37944-6808-42f4-857f-57ea8635a2cd","Type":"ContainerStarted","Data":"c785d0aa0ff0c3d6466d164c83fd405c954295e858f561dc337e55658650c6cd"} Nov 03 22:45:01.904065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.904040 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" event={"ID":"f3056f50-50c4-4804-af55-0e9e4f06f583","Type":"ContainerStarted","Data":"d7ca4bf48b3b757e975d4712a0b454844d982152995c516866a32136c1acba03"} Nov 03 22:45:01.906106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.906078 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" event={"ID":"22571f32-7943-4e26-96f5-f409490f2969","Type":"ContainerStarted","Data":"85ce9746023a152db14c04334548e7e9470731318f45eb67735f344847ac211e"} Nov 03 22:45:01.906233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.906128 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" event={"ID":"22571f32-7943-4e26-96f5-f409490f2969","Type":"ContainerStarted","Data":"8189f375f9d5a3a906d3df3b68d90b75615bb3321c10bcd0264ed9881ec690c7"} Nov 03 22:45:01.908010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.907984 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" event={"ID":"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12","Type":"ContainerStarted","Data":"578dc4d9be2855b27b3a390d105b5da5e438f0d52ba754ec845667dd78091a48"} Nov 03 22:45:01.908129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.908035 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" event={"ID":"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12","Type":"ContainerStarted","Data":"81639ad591033d5bfabe59af3e4fa23236bf45bd6294c047761b9a2f04f72ad5"} Nov 03 22:45:01.909802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.909770 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" event={"ID":"50000cc4-9bd6-4cfb-a85f-15b0ae191071","Type":"ContainerStarted","Data":"da897054523724e1084a42422e2d25bb09d70f5c93bd040459c10ec656b5964f"} Nov 03 22:45:01.909927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.909819 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" event={"ID":"50000cc4-9bd6-4cfb-a85f-15b0ae191071","Type":"ContainerStarted","Data":"c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c"} Nov 03 22:45:01.911561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.911511 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" event={"ID":"157ee087-43db-4f2f-879d-a6e477ddf041","Type":"ContainerStarted","Data":"a6393e97ffba37711308582b034c7e4ef5c90864bdf63c434de605b020edc418"} Nov 03 22:45:01.913202 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.913180 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" event={"ID":"81f863b3-9457-4888-b046-3cfd408e504a","Type":"ContainerStarted","Data":"d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef"} Nov 03 22:45:01.914525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.914497 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" event={"ID":"c029cafa-94ab-4b6a-b775-3fee6c1acea6","Type":"ContainerStarted","Data":"d150b45413534c29dfe15b12fdaab333b0bec9086468d51ffdee4807d4f0a779"} Nov 03 22:45:01.916421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.916220 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" event={"ID":"1534c655-737d-499d-8fcf-00e85c96854c","Type":"ContainerStarted","Data":"77b7e55fa0781cb3be5e2031af8011dd9612f1abf525dca5b497b89da4389ec0"} Nov 03 22:45:01.918018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:01.917995 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" event={"ID":"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17","Type":"ContainerStarted","Data":"ba86baabbac423d6c70ce421b2f48bfffe0103be51c36275f65182e2c13168ce"} Nov 03 22:45:02.033885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.029021 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" podStartSLOduration=2.029002085 podStartE2EDuration="2.029002085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:02.021828432 +0000 UTC m=+2259.002562084" watchObservedRunningTime="2026-11-03 22:45:02.029002085 +0000 UTC m=+2259.009735738" Nov 03 22:45:02.139779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.139716 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" podStartSLOduration=2.139696294 podStartE2EDuration="2.139696294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:02.135078159 +0000 UTC m=+2259.115811805" watchObservedRunningTime="2026-11-03 22:45:02.139696294 +0000 UTC m=+2259.120429949" Nov 03 22:45:02.253257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.253205 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" podStartSLOduration=2.253188231 podStartE2EDuration="2.253188231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:02.250879571 +0000 UTC m=+2259.231613225" watchObservedRunningTime="2026-11-03 22:45:02.253188231 +0000 UTC m=+2259.233921883" Nov 03 22:45:02.483204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.483177 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:02.834884 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.831196 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" containerID="cri-o://990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a" gracePeriod=45 Nov 03 22:45:02.936366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.936320 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" event={"ID":"1534c655-737d-499d-8fcf-00e85c96854c","Type":"ContainerStarted","Data":"cb65a4a26c4350918f74304844a1280b0db57245f1b7f4a68de0e00f494f7639"} Nov 03 22:45:02.939140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.939101 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" event={"ID":"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17","Type":"ContainerStarted","Data":"2bff0df007725998f2165b72f0947fd814c6b92fb1abfd5c089a9befb9d76954"} Nov 03 22:45:02.941783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.941721 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" event={"ID":"ecc2af8c-c3b6-4bde-ac6f-398533e20acf","Type":"ContainerStarted","Data":"82d846862891efca0e072c50c9b6d573b45b12e7e0be5b1d52064ab43919869d"} Nov 03 22:45:02.943925 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.943894 4298 generic.go:356] "Generic (PLEG): container finished" podID="03b37944-6808-42f4-857f-57ea8635a2cd" containerID="1dfac259669927ed981f30b762238c074dccb1714db18b744e322ee040f272d8" exitCode=60 Nov 03 22:45:02.944082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.943984 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" event={"ID":"03b37944-6808-42f4-857f-57ea8635a2cd","Type":"ContainerDied","Data":"1dfac259669927ed981f30b762238c074dccb1714db18b744e322ee040f272d8"} Nov 03 22:45:02.944244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.944223 4298 scope.go:122] "RemoveContainer" containerID="1dfac259669927ed981f30b762238c074dccb1714db18b744e322ee040f272d8" Nov 03 22:45:02.946054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.946030 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" event={"ID":"f3056f50-50c4-4804-af55-0e9e4f06f583","Type":"ContainerStarted","Data":"df30236c9e06c1bfbcc4d898fb72c97806d71ec9cd9cda7af75bb445ae148d26"} Nov 03 22:45:02.946428 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.946383 4298 scope.go:122] "RemoveContainer" containerID="df30236c9e06c1bfbcc4d898fb72c97806d71ec9cd9cda7af75bb445ae148d26" Nov 03 22:45:02.956502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.956458 4298 generic.go:356] "Generic (PLEG): container finished" podID="22571f32-7943-4e26-96f5-f409490f2969" containerID="85ce9746023a152db14c04334548e7e9470731318f45eb67735f344847ac211e" exitCode=60 Nov 03 22:45:02.956677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.956496 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" event={"ID":"22571f32-7943-4e26-96f5-f409490f2969","Type":"ContainerDied","Data":"85ce9746023a152db14c04334548e7e9470731318f45eb67735f344847ac211e"} Nov 03 22:45:02.956883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.956865 4298 scope.go:122] "RemoveContainer" containerID="85ce9746023a152db14c04334548e7e9470731318f45eb67735f344847ac211e" Nov 03 22:45:02.961104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.961075 4298 generic.go:356] "Generic (PLEG): container finished" podID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerID="578dc4d9be2855b27b3a390d105b5da5e438f0d52ba754ec845667dd78091a48" exitCode=60 Nov 03 22:45:02.961456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.961384 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" event={"ID":"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12","Type":"ContainerDied","Data":"578dc4d9be2855b27b3a390d105b5da5e438f0d52ba754ec845667dd78091a48"} Nov 03 22:45:02.961571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.961510 4298 scope.go:122] "RemoveContainer" containerID="578dc4d9be2855b27b3a390d105b5da5e438f0d52ba754ec845667dd78091a48" Nov 03 22:45:02.964067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.963436 4298 generic.go:356] "Generic (PLEG): container finished" podID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerID="da897054523724e1084a42422e2d25bb09d70f5c93bd040459c10ec656b5964f" exitCode=60 Nov 03 22:45:02.964067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.963692 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" event={"ID":"50000cc4-9bd6-4cfb-a85f-15b0ae191071","Type":"ContainerDied","Data":"da897054523724e1084a42422e2d25bb09d70f5c93bd040459c10ec656b5964f"} Nov 03 22:45:02.964067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.963776 4298 scope.go:122] "RemoveContainer" containerID="da897054523724e1084a42422e2d25bb09d70f5c93bd040459c10ec656b5964f" Nov 03 22:45:02.966986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.966953 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" event={"ID":"157ee087-43db-4f2f-879d-a6e477ddf041","Type":"ContainerStarted","Data":"89028362b3b757dd52b7a2fc4e326986a940e30382560814edf63c13ac17d0bf"} Nov 03 22:45:02.971687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.971601 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" event={"ID":"81f863b3-9457-4888-b046-3cfd408e504a","Type":"ContainerStarted","Data":"3a69d5e81c4c18d8017e06df3b37b4797f058602684dcbfbb921e2a69765e45a"} Nov 03 22:45:02.974126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.974101 4298 generic.go:356] "Generic (PLEG): container finished" podID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerID="23d634c834b9c4ab34eb950f3d927e73253f97b5cc79a77f8407bcd08541c948" exitCode=60 Nov 03 22:45:02.974213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.974148 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" event={"ID":"c029cafa-94ab-4b6a-b775-3fee6c1acea6","Type":"ContainerDied","Data":"23d634c834b9c4ab34eb950f3d927e73253f97b5cc79a77f8407bcd08541c948"} Nov 03 22:45:02.974551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:02.974528 4298 scope.go:122] "RemoveContainer" containerID="23d634c834b9c4ab34eb950f3d927e73253f97b5cc79a77f8407bcd08541c948" Nov 03 22:45:03.069247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.069195 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" podStartSLOduration=3.069159133 podStartE2EDuration="3.069159133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:03.049474266 +0000 UTC m=+2260.030207920" watchObservedRunningTime="2026-11-03 22:45:03.069159133 +0000 UTC m=+2260.049892786" Nov 03 22:45:03.348146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.348117 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:45:03.469338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.469293 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tmp-dir\") pod \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " Nov 03 22:45:03.469338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.469341 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-home\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-home\") pod \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " Nov 03 22:45:03.469610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.469433 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tls-certs\") pod \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " Nov 03 22:45:03.469610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.469468 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-kube-api-access-gmzwt\" (UniqueName: \"kubernetes.io/projected/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-kube-api-access-gmzwt\") pod \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " Nov 03 22:45:03.469610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.469519 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-model-cache\") pod \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " Nov 03 22:45:03.469610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.469595 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-dshm\") pod \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\" (UID: \"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0\") " Nov 03 22:45:03.469841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.469717 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-home" pod "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" (UID: "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.469981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.469953 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-model-cache" pod "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" (UID: "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.470065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.469735 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tmp-dir" pod "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" (UID: "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.470274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.470256 4298 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tmp-dir\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.470274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.470274 4298 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-home\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.470427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.470282 4298 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-model-cache\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.530431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.530385 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-dshm" pod "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" (UID: "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.530591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.530441 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-kube-api-access-gmzwt" pod "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" (UID: "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0"). InnerVolumeSpecName "kube-api-access-gmzwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:03.530691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.530405 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tls-certs" pod "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" (UID: "4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:03.571827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.571792 4298 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-tls-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.571827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.571824 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmzwt\" (UniqueName: \"kubernetes.io/projected/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-kube-api-access-gmzwt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.571827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.571836 4298 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0-dshm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.623533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.623479 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" podStartSLOduration=3.623462434 podStartE2EDuration="3.623462434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:03.621156553 +0000 UTC m=+2260.601890207" watchObservedRunningTime="2026-11-03 22:45:03.623462434 +0000 UTC m=+2260.604196124" Nov 03 22:45:03.737871 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.737812 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" podStartSLOduration=3.737789793 podStartE2EDuration="3.737789793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:03.734237923 +0000 UTC m=+2260.714971573" watchObservedRunningTime="2026-11-03 22:45:03.737789793 +0000 UTC m=+2260.718523447" Nov 03 22:45:03.985694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.985623 4298 generic.go:356] "Generic (PLEG): container finished" podID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerID="82d846862891efca0e072c50c9b6d573b45b12e7e0be5b1d52064ab43919869d" exitCode=60 Nov 03 22:45:03.985694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.985678 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" event={"ID":"ecc2af8c-c3b6-4bde-ac6f-398533e20acf","Type":"ContainerDied","Data":"82d846862891efca0e072c50c9b6d573b45b12e7e0be5b1d52064ab43919869d"} Nov 03 22:45:03.986121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.986097 4298 scope.go:122] "RemoveContainer" containerID="82d846862891efca0e072c50c9b6d573b45b12e7e0be5b1d52064ab43919869d" Nov 03 22:45:03.987835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.987813 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" event={"ID":"03b37944-6808-42f4-857f-57ea8635a2cd","Type":"ContainerStarted","Data":"689d936d6f0d58c966c8f8df944319bc45bb7a44e8ef2a303769fb8e3e1ed3de"} Nov 03 22:45:03.990117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.990095 4298 generic.go:356] "Generic (PLEG): container finished" podID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerID="df30236c9e06c1bfbcc4d898fb72c97806d71ec9cd9cda7af75bb445ae148d26" exitCode=60 Nov 03 22:45:03.990183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.990149 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" event={"ID":"f3056f50-50c4-4804-af55-0e9e4f06f583","Type":"ContainerDied","Data":"df30236c9e06c1bfbcc4d898fb72c97806d71ec9cd9cda7af75bb445ae148d26"} Nov 03 22:45:03.990230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.990200 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" event={"ID":"f3056f50-50c4-4804-af55-0e9e4f06f583","Type":"ContainerStarted","Data":"bc9440ee56644aebb4dc7e46bd98585c2e2c95169be7ed3e6d0a47ad6c2b493f"} Nov 03 22:45:03.992253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.992226 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" event={"ID":"22571f32-7943-4e26-96f5-f409490f2969","Type":"ContainerStarted","Data":"b5a2bf7dde5cd1ab1c0a37b0de412fc12b170a6a100dc458aa8d66a3f2a9e4e2"} Nov 03 22:45:03.994151 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.994130 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" event={"ID":"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12","Type":"ContainerStarted","Data":"92c91eb93dd61f8a46d14dfcbb7dde1140f3c889d8c6b147deebd5ac2f1d89e7"} Nov 03 22:45:03.996149 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.996126 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" event={"ID":"50000cc4-9bd6-4cfb-a85f-15b0ae191071","Type":"ContainerStarted","Data":"5e231792f7eec04ca45d8193fe9ca71b036f6729ab844dab7021e79646c2085a"} Nov 03 22:45:03.997457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.997431 4298 generic.go:356] "Generic (PLEG): container finished" podID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerID="990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a" exitCode=0 Nov 03 22:45:03.997547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.997479 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" Nov 03 22:45:03.997547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.997506 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" event={"ID":"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0","Type":"ContainerDied","Data":"990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a"} Nov 03 22:45:03.997692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.997545 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x" event={"ID":"4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0","Type":"ContainerDied","Data":"1d6a344b144e4950202f1acd0882aa97e6e28504e386c710fef8c7a3c667b88e"} Nov 03 22:45:03.997692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.997566 4298 scope.go:122] "RemoveContainer" containerID="990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a" Nov 03 22:45:03.999394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.999366 4298 generic.go:356] "Generic (PLEG): container finished" podID="157ee087-43db-4f2f-879d-a6e477ddf041" containerID="89028362b3b757dd52b7a2fc4e326986a940e30382560814edf63c13ac17d0bf" exitCode=60 Nov 03 22:45:03.999638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.999614 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" event={"ID":"157ee087-43db-4f2f-879d-a6e477ddf041","Type":"ContainerDied","Data":"89028362b3b757dd52b7a2fc4e326986a940e30382560814edf63c13ac17d0bf"} Nov 03 22:45:03.999929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:03.999912 4298 scope.go:122] "RemoveContainer" containerID="89028362b3b757dd52b7a2fc4e326986a940e30382560814edf63c13ac17d0bf" Nov 03 22:45:04.001420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.001397 4298 generic.go:356] "Generic (PLEG): container finished" podID="81f863b3-9457-4888-b046-3cfd408e504a" containerID="3a69d5e81c4c18d8017e06df3b37b4797f058602684dcbfbb921e2a69765e45a" exitCode=60 Nov 03 22:45:04.001960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.001541 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" event={"ID":"81f863b3-9457-4888-b046-3cfd408e504a","Type":"ContainerDied","Data":"3a69d5e81c4c18d8017e06df3b37b4797f058602684dcbfbb921e2a69765e45a"} Nov 03 22:45:04.001960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.001723 4298 scope.go:122] "RemoveContainer" containerID="3a69d5e81c4c18d8017e06df3b37b4797f058602684dcbfbb921e2a69765e45a" Nov 03 22:45:04.005922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.005894 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" event={"ID":"c029cafa-94ab-4b6a-b775-3fee6c1acea6","Type":"ContainerStarted","Data":"b5cf9760a8b540d46859dec27af90bcf463f7379f61c65374a2ddfc40fa437b2"} Nov 03 22:45:04.008434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.008408 4298 generic.go:356] "Generic (PLEG): container finished" podID="1534c655-737d-499d-8fcf-00e85c96854c" containerID="cb65a4a26c4350918f74304844a1280b0db57245f1b7f4a68de0e00f494f7639" exitCode=60 Nov 03 22:45:04.008523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.008486 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" event={"ID":"1534c655-737d-499d-8fcf-00e85c96854c","Type":"ContainerDied","Data":"cb65a4a26c4350918f74304844a1280b0db57245f1b7f4a68de0e00f494f7639"} Nov 03 22:45:04.008713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.008699 4298 scope.go:122] "RemoveContainer" containerID="cb65a4a26c4350918f74304844a1280b0db57245f1b7f4a68de0e00f494f7639" Nov 03 22:45:04.010309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.010290 4298 generic.go:356] "Generic (PLEG): container finished" podID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerID="2bff0df007725998f2165b72f0947fd814c6b92fb1abfd5c089a9befb9d76954" exitCode=60 Nov 03 22:45:04.010400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.010327 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" event={"ID":"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17","Type":"ContainerDied","Data":"2bff0df007725998f2165b72f0947fd814c6b92fb1abfd5c089a9befb9d76954"} Nov 03 22:45:04.010599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.010578 4298 scope.go:122] "RemoveContainer" containerID="2bff0df007725998f2165b72f0947fd814c6b92fb1abfd5c089a9befb9d76954" Nov 03 22:45:04.011215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.011199 4298 scope.go:122] "RemoveContainer" containerID="990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a" Nov 03 22:45:04.011528 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:04.011507 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a\": container with ID starting with 990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a not found: ID does not exist" containerID="990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a" Nov 03 22:45:04.011602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.011540 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a"} err="failed to get container status \"990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a\": rpc error: code = NotFound desc = could not find container \"990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a\": container with ID starting with 990d1222ab31578a9c8f7d88349b781702ed306a46ce417abb6be4ff20be8c7a not found: ID does not exist" Nov 03 22:45:04.076045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.075992 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" podStartSLOduration=4.075973895 podStartE2EDuration="4.075973895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:04.073347112 +0000 UTC m=+2261.054080767" watchObservedRunningTime="2026-11-03 22:45:04.075973895 +0000 UTC m=+2261.056707561" Nov 03 22:45:04.189113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.189000 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" podStartSLOduration=4.188984267 podStartE2EDuration="4.188984267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:04.188187498 +0000 UTC m=+2261.168921153" watchObservedRunningTime="2026-11-03 22:45:04.188984267 +0000 UTC m=+2261.169717919" Nov 03 22:45:04.371638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.371607 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x"] Nov 03 22:45:04.382679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.382606 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-2-w1-3378fd-kserve-5c974f5d6d-4gg6x"] Nov 03 22:45:04.607201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.607137 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" podStartSLOduration=4.607121171 podStartE2EDuration="4.607121171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:04.60207906 +0000 UTC m=+2261.582812714" watchObservedRunningTime="2026-11-03 22:45:04.607121171 +0000 UTC m=+2261.587854861" Nov 03 22:45:04.959595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:04.959538 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" podStartSLOduration=4.959522382 podStartE2EDuration="4.959522382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:04.956032635 +0000 UTC m=+2261.936766287" watchObservedRunningTime="2026-11-03 22:45:04.959522382 +0000 UTC m=+2261.940256034" Nov 03 22:45:05.020886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.020851 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" event={"ID":"ecc2af8c-c3b6-4bde-ac6f-398533e20acf","Type":"ContainerStarted","Data":"31cf87aebe6e0fa1d5bfe6bd5ecafbe8f5a22afbec4d57754c334901638b4747"} Nov 03 22:45:05.023190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.023158 4298 generic.go:356] "Generic (PLEG): container finished" podID="03b37944-6808-42f4-857f-57ea8635a2cd" containerID="689d936d6f0d58c966c8f8df944319bc45bb7a44e8ef2a303769fb8e3e1ed3de" exitCode=60 Nov 03 22:45:05.023359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.023256 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" event={"ID":"03b37944-6808-42f4-857f-57ea8635a2cd","Type":"ContainerDied","Data":"689d936d6f0d58c966c8f8df944319bc45bb7a44e8ef2a303769fb8e3e1ed3de"} Nov 03 22:45:05.023359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.023315 4298 scope.go:122] "RemoveContainer" containerID="1dfac259669927ed981f30b762238c074dccb1714db18b744e322ee040f272d8" Nov 03 22:45:05.023637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.023617 4298 scope.go:122] "RemoveContainer" containerID="689d936d6f0d58c966c8f8df944319bc45bb7a44e8ef2a303769fb8e3e1ed3de" Nov 03 22:45:05.024073 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:05.024050 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895765-vbght_odh-ai-gateway-infra(03b37944-6808-42f4-857f-57ea8635a2cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" Nov 03 22:45:05.030491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.030403 4298 generic.go:356] "Generic (PLEG): container finished" podID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerID="bc9440ee56644aebb4dc7e46bd98585c2e2c95169be7ed3e6d0a47ad6c2b493f" exitCode=60 Nov 03 22:45:05.030491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.030470 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" event={"ID":"f3056f50-50c4-4804-af55-0e9e4f06f583","Type":"ContainerDied","Data":"bc9440ee56644aebb4dc7e46bd98585c2e2c95169be7ed3e6d0a47ad6c2b493f"} Nov 03 22:45:05.030911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.030889 4298 scope.go:122] "RemoveContainer" containerID="bc9440ee56644aebb4dc7e46bd98585c2e2c95169be7ed3e6d0a47ad6c2b493f" Nov 03 22:45:05.031238 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:05.031220 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp_odh-ai-gateway-infra(f3056f50-50c4-4804-af55-0e9e4f06f583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" Nov 03 22:45:05.033319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.033296 4298 generic.go:356] "Generic (PLEG): container finished" podID="22571f32-7943-4e26-96f5-f409490f2969" containerID="b5a2bf7dde5cd1ab1c0a37b0de412fc12b170a6a100dc458aa8d66a3f2a9e4e2" exitCode=60 Nov 03 22:45:05.033424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.033371 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" event={"ID":"22571f32-7943-4e26-96f5-f409490f2969","Type":"ContainerDied","Data":"b5a2bf7dde5cd1ab1c0a37b0de412fc12b170a6a100dc458aa8d66a3f2a9e4e2"} Nov 03 22:45:05.033667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.033623 4298 scope.go:122] "RemoveContainer" containerID="b5a2bf7dde5cd1ab1c0a37b0de412fc12b170a6a100dc458aa8d66a3f2a9e4e2" Nov 03 22:45:05.034291 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:05.034271 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b_odh-ai-gateway-infra(22571f32-7943-4e26-96f5-f409490f2969)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" podUID="22571f32-7943-4e26-96f5-f409490f2969" Nov 03 22:45:05.035796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.035771 4298 generic.go:356] "Generic (PLEG): container finished" podID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerID="92c91eb93dd61f8a46d14dfcbb7dde1140f3c889d8c6b147deebd5ac2f1d89e7" exitCode=60 Nov 03 22:45:05.035902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.035854 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" event={"ID":"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12","Type":"ContainerDied","Data":"92c91eb93dd61f8a46d14dfcbb7dde1140f3c889d8c6b147deebd5ac2f1d89e7"} Nov 03 22:45:05.036257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.036158 4298 scope.go:122] "RemoveContainer" containerID="92c91eb93dd61f8a46d14dfcbb7dde1140f3c889d8c6b147deebd5ac2f1d89e7" Nov 03 22:45:05.036517 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:05.036486 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz_odh-ai-gateway-infra(21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" Nov 03 22:45:05.038385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.038363 4298 generic.go:356] "Generic (PLEG): container finished" podID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerID="5e231792f7eec04ca45d8193fe9ca71b036f6729ab844dab7021e79646c2085a" exitCode=60 Nov 03 22:45:05.038492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.038413 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" event={"ID":"50000cc4-9bd6-4cfb-a85f-15b0ae191071","Type":"ContainerDied","Data":"5e231792f7eec04ca45d8193fe9ca71b036f6729ab844dab7021e79646c2085a"} Nov 03 22:45:05.038721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.038699 4298 scope.go:122] "RemoveContainer" containerID="5e231792f7eec04ca45d8193fe9ca71b036f6729ab844dab7021e79646c2085a" Nov 03 22:45:05.039022 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:05.039000 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn_odh-ai-gateway-infra(50000cc4-9bd6-4cfb-a85f-15b0ae191071)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" Nov 03 22:45:05.043697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.043673 4298 scope.go:122] "RemoveContainer" containerID="df30236c9e06c1bfbcc4d898fb72c97806d71ec9cd9cda7af75bb445ae148d26" Nov 03 22:45:05.043873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.043846 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" event={"ID":"157ee087-43db-4f2f-879d-a6e477ddf041","Type":"ContainerStarted","Data":"c6d9f2797bb366d9a5d73ad03637d994772e68474d815ff2e4d8d3edfbc4402d"} Nov 03 22:45:05.046075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.046051 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" event={"ID":"81f863b3-9457-4888-b046-3cfd408e504a","Type":"ContainerStarted","Data":"1d838ccdc93e162f1102f4a17880babb5cd0e8766abbd58b5a2e077bb0f62feb"} Nov 03 22:45:05.048228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.048203 4298 generic.go:356] "Generic (PLEG): container finished" podID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerID="b5cf9760a8b540d46859dec27af90bcf463f7379f61c65374a2ddfc40fa437b2" exitCode=60 Nov 03 22:45:05.048350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.048280 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" event={"ID":"c029cafa-94ab-4b6a-b775-3fee6c1acea6","Type":"ContainerDied","Data":"b5cf9760a8b540d46859dec27af90bcf463f7379f61c65374a2ddfc40fa437b2"} Nov 03 22:45:05.048583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.048566 4298 scope.go:122] "RemoveContainer" containerID="b5cf9760a8b540d46859dec27af90bcf463f7379f61c65374a2ddfc40fa437b2" Nov 03 22:45:05.048926 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:05.048905 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6_odh-ai-gateway-infra(c029cafa-94ab-4b6a-b775-3fee6c1acea6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" Nov 03 22:45:05.050475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.050451 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" event={"ID":"1534c655-737d-499d-8fcf-00e85c96854c","Type":"ContainerStarted","Data":"020aef95d21d12842fde0d134dea4565b6ddae6ca60db3c05a4f1a25950e1db0"} Nov 03 22:45:05.052730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.052706 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" event={"ID":"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17","Type":"ContainerStarted","Data":"ed0c91736c511c3d345830b440ff0aab7c7fee783a912afcd92c7849a37b469b"} Nov 03 22:45:05.102368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.102347 4298 scope.go:122] "RemoveContainer" containerID="85ce9746023a152db14c04334548e7e9470731318f45eb67735f344847ac211e" Nov 03 22:45:05.116503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.116475 4298 scope.go:122] "RemoveContainer" containerID="578dc4d9be2855b27b3a390d105b5da5e438f0d52ba754ec845667dd78091a48" Nov 03 22:45:05.135031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.133999 4298 scope.go:122] "RemoveContainer" containerID="da897054523724e1084a42422e2d25bb09d70f5c93bd040459c10ec656b5964f" Nov 03 22:45:05.152268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.152235 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" path="/var/lib/kubelet/pods/4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0/volumes" Nov 03 22:45:05.409043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.408915 4298 scope.go:122] "RemoveContainer" containerID="23d634c834b9c4ab34eb950f3d927e73253f97b5cc79a77f8407bcd08541c948" Nov 03 22:45:05.416273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.416221 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" podStartSLOduration=5.4162067799999996 podStartE2EDuration="5.41620678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:05.412205039 +0000 UTC m=+2262.392938691" watchObservedRunningTime="2026-11-03 22:45:05.41620678 +0000 UTC m=+2262.396940434" Nov 03 22:45:05.954350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:05.954310 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:06.063146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.063106 4298 generic.go:356] "Generic (PLEG): container finished" podID="157ee087-43db-4f2f-879d-a6e477ddf041" containerID="c6d9f2797bb366d9a5d73ad03637d994772e68474d815ff2e4d8d3edfbc4402d" exitCode=60 Nov 03 22:45:06.063343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.063188 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" event={"ID":"157ee087-43db-4f2f-879d-a6e477ddf041","Type":"ContainerDied","Data":"c6d9f2797bb366d9a5d73ad03637d994772e68474d815ff2e4d8d3edfbc4402d"} Nov 03 22:45:06.063343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.063261 4298 scope.go:122] "RemoveContainer" containerID="89028362b3b757dd52b7a2fc4e326986a940e30382560814edf63c13ac17d0bf" Nov 03 22:45:06.063567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.063550 4298 scope.go:122] "RemoveContainer" containerID="c6d9f2797bb366d9a5d73ad03637d994772e68474d815ff2e4d8d3edfbc4402d" Nov 03 22:45:06.063967 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:06.063941 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr_odh-ai-gateway-infra(157ee087-43db-4f2f-879d-a6e477ddf041)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" Nov 03 22:45:06.065263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.065236 4298 generic.go:356] "Generic (PLEG): container finished" podID="81f863b3-9457-4888-b046-3cfd408e504a" containerID="1d838ccdc93e162f1102f4a17880babb5cd0e8766abbd58b5a2e077bb0f62feb" exitCode=60 Nov 03 22:45:06.065420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.065269 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" event={"ID":"81f863b3-9457-4888-b046-3cfd408e504a","Type":"ContainerDied","Data":"1d838ccdc93e162f1102f4a17880babb5cd0e8766abbd58b5a2e077bb0f62feb"} Nov 03 22:45:06.065578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.065564 4298 scope.go:122] "RemoveContainer" containerID="1d838ccdc93e162f1102f4a17880babb5cd0e8766abbd58b5a2e077bb0f62feb" Nov 03 22:45:06.065943 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:06.065919 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr_odh-ai-gateway-infra(81f863b3-9457-4888-b046-3cfd408e504a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" podUID="81f863b3-9457-4888-b046-3cfd408e504a" Nov 03 22:45:06.067250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.067235 4298 scope.go:122] "RemoveContainer" containerID="b5cf9760a8b540d46859dec27af90bcf463f7379f61c65374a2ddfc40fa437b2" Nov 03 22:45:06.067540 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:06.067523 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6_odh-ai-gateway-infra(c029cafa-94ab-4b6a-b775-3fee6c1acea6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" Nov 03 22:45:06.069052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.069032 4298 generic.go:356] "Generic (PLEG): container finished" podID="1534c655-737d-499d-8fcf-00e85c96854c" containerID="020aef95d21d12842fde0d134dea4565b6ddae6ca60db3c05a4f1a25950e1db0" exitCode=60 Nov 03 22:45:06.069157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.069110 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" event={"ID":"1534c655-737d-499d-8fcf-00e85c96854c","Type":"ContainerDied","Data":"020aef95d21d12842fde0d134dea4565b6ddae6ca60db3c05a4f1a25950e1db0"} Nov 03 22:45:06.069363 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.069345 4298 scope.go:122] "RemoveContainer" containerID="020aef95d21d12842fde0d134dea4565b6ddae6ca60db3c05a4f1a25950e1db0" Nov 03 22:45:06.069683 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:06.069658 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd_odh-ai-gateway-infra(1534c655-737d-499d-8fcf-00e85c96854c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" podUID="1534c655-737d-499d-8fcf-00e85c96854c" Nov 03 22:45:06.071104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.071079 4298 generic.go:356] "Generic (PLEG): container finished" podID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerID="ed0c91736c511c3d345830b440ff0aab7c7fee783a912afcd92c7849a37b469b" exitCode=60 Nov 03 22:45:06.071193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.071150 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" event={"ID":"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17","Type":"ContainerDied","Data":"ed0c91736c511c3d345830b440ff0aab7c7fee783a912afcd92c7849a37b469b"} Nov 03 22:45:06.071361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.071337 4298 scope.go:122] "RemoveContainer" containerID="ed0c91736c511c3d345830b440ff0aab7c7fee783a912afcd92c7849a37b469b" Nov 03 22:45:06.071605 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:06.071583 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6_odh-ai-gateway-infra(9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" Nov 03 22:45:06.073242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.073213 4298 generic.go:356] "Generic (PLEG): container finished" podID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerID="31cf87aebe6e0fa1d5bfe6bd5ecafbe8f5a22afbec4d57754c334901638b4747" exitCode=60 Nov 03 22:45:06.073346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.073286 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" event={"ID":"ecc2af8c-c3b6-4bde-ac6f-398533e20acf","Type":"ContainerDied","Data":"31cf87aebe6e0fa1d5bfe6bd5ecafbe8f5a22afbec4d57754c334901638b4747"} Nov 03 22:45:06.073507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.073492 4298 scope.go:122] "RemoveContainer" containerID="31cf87aebe6e0fa1d5bfe6bd5ecafbe8f5a22afbec4d57754c334901638b4747" Nov 03 22:45:06.073835 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:06.073813 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6_odh-ai-gateway-infra(ecc2af8c-c3b6-4bde-ac6f-398533e20acf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" Nov 03 22:45:06.075348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.075331 4298 scope.go:122] "RemoveContainer" containerID="689d936d6f0d58c966c8f8df944319bc45bb7a44e8ef2a303769fb8e3e1ed3de" Nov 03 22:45:06.075572 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:06.075549 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895765-vbght_odh-ai-gateway-infra(03b37944-6808-42f4-857f-57ea8635a2cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" Nov 03 22:45:06.076922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.076902 4298 scope.go:122] "RemoveContainer" containerID="bc9440ee56644aebb4dc7e46bd98585c2e2c95169be7ed3e6d0a47ad6c2b493f" Nov 03 22:45:06.077123 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:06.077101 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp_odh-ai-gateway-infra(f3056f50-50c4-4804-af55-0e9e4f06f583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" Nov 03 22:45:06.078334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.078314 4298 generic.go:356] "Generic (PLEG): container finished" podID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerID="87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413" exitCode=0 Nov 03 22:45:06.078424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.078379 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" event={"ID":"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2","Type":"ContainerDied","Data":"87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413"} Nov 03 22:45:06.078744 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.078726 4298 scope.go:122] "RemoveContainer" containerID="87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413" Nov 03 22:45:06.079335 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.079306 4298 scope.go:122] "RemoveContainer" containerID="3a69d5e81c4c18d8017e06df3b37b4797f058602684dcbfbb921e2a69765e45a" Nov 03 22:45:06.080257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.080240 4298 scope.go:122] "RemoveContainer" containerID="b5a2bf7dde5cd1ab1c0a37b0de412fc12b170a6a100dc458aa8d66a3f2a9e4e2" Nov 03 22:45:06.080419 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.080315 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:06.080468 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.080424 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:06.080518 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.080473 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:06.080572 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:06.080516 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b_odh-ai-gateway-infra(22571f32-7943-4e26-96f5-f409490f2969)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" podUID="22571f32-7943-4e26-96f5-f409490f2969" Nov 03 22:45:06.081932 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.081913 4298 scope.go:122] "RemoveContainer" containerID="92c91eb93dd61f8a46d14dfcbb7dde1140f3c889d8c6b147deebd5ac2f1d89e7" Nov 03 22:45:06.082785 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:06.082762 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz_odh-ai-gateway-infra(21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" Nov 03 22:45:06.083398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.083381 4298 scope.go:122] "RemoveContainer" containerID="5e231792f7eec04ca45d8193fe9ca71b036f6729ab844dab7021e79646c2085a" Nov 03 22:45:06.083586 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:06.083566 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn_odh-ai-gateway-infra(50000cc4-9bd6-4cfb-a85f-15b0ae191071)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" Nov 03 22:45:06.095630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.095603 4298 scope.go:122] "RemoveContainer" containerID="cb65a4a26c4350918f74304844a1280b0db57245f1b7f4a68de0e00f494f7639" Nov 03 22:45:06.121151 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.121126 4298 scope.go:122] "RemoveContainer" containerID="2bff0df007725998f2165b72f0947fd814c6b92fb1abfd5c089a9befb9d76954" Nov 03 22:45:06.136432 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:06.136413 4298 scope.go:122] "RemoveContainer" containerID="82d846862891efca0e072c50c9b6d573b45b12e7e0be5b1d52064ab43919869d" Nov 03 22:45:07.123903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:07.121570 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" event={"ID":"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2","Type":"ContainerStarted","Data":"4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8"} Nov 03 22:45:07.136672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:07.127276 4298 scope.go:122] "RemoveContainer" containerID="c6d9f2797bb366d9a5d73ad03637d994772e68474d815ff2e4d8d3edfbc4402d" Nov 03 22:45:07.136672 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:07.127725 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr_odh-ai-gateway-infra(157ee087-43db-4f2f-879d-a6e477ddf041)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" Nov 03 22:45:07.136672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:07.135814 4298 scope.go:122] "RemoveContainer" containerID="1d838ccdc93e162f1102f4a17880babb5cd0e8766abbd58b5a2e077bb0f62feb" Nov 03 22:45:07.136672 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:07.136225 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr_odh-ai-gateway-infra(81f863b3-9457-4888-b046-3cfd408e504a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" podUID="81f863b3-9457-4888-b046-3cfd408e504a" Nov 03 22:45:07.147177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:07.139102 4298 scope.go:122] "RemoveContainer" containerID="020aef95d21d12842fde0d134dea4565b6ddae6ca60db3c05a4f1a25950e1db0" Nov 03 22:45:07.147177 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:07.139470 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd_odh-ai-gateway-infra(1534c655-737d-499d-8fcf-00e85c96854c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" podUID="1534c655-737d-499d-8fcf-00e85c96854c" Nov 03 22:45:07.147177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:07.142331 4298 scope.go:122] "RemoveContainer" containerID="ed0c91736c511c3d345830b440ff0aab7c7fee783a912afcd92c7849a37b469b" Nov 03 22:45:07.147177 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:07.142712 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6_odh-ai-gateway-infra(9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" Nov 03 22:45:07.154667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:07.150534 4298 scope.go:122] "RemoveContainer" containerID="31cf87aebe6e0fa1d5bfe6bd5ecafbe8f5a22afbec4d57754c334901638b4747" Nov 03 22:45:07.154667 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:07.150912 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6_odh-ai-gateway-infra(ecc2af8c-c3b6-4bde-ac6f-398533e20acf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" Nov 03 22:45:07.380250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:07.380152 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:07.647729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:07.641558 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:45:07.651081 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:07.651039 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:45:07.652100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:07.652076 4298 patch_prober.go:29] interesting pod/e2e-api-a-c8f168-openshift-default-786989b665-shc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:45:07.652236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:07.652136 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 22:45:08.651229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:08.651186 4298 patch_prober.go:29] interesting pod/e2e-api-a-c8f168-openshift-default-786989b665-shc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:45:08.651681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:08.651260 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 22:45:09.651220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:09.651183 4298 patch_prober.go:29] interesting pod/e2e-api-a-c8f168-openshift-default-786989b665-shc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:45:09.651407 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:09.651242 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 22:45:10.651519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:10.651488 4298 patch_prober.go:29] interesting pod/e2e-api-a-c8f168-openshift-default-786989b665-shc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:45:10.651931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:10.651540 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 22:45:11.662672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:11.660006 4298 patch_prober.go:29] interesting pod/e2e-api-a-c8f168-openshift-default-786989b665-shc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:45:11.662672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:11.660069 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 22:45:12.408022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:12.407991 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:12.651549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:12.651517 4298 patch_prober.go:29] interesting pod/e2e-api-a-c8f168-openshift-default-786989b665-shc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:45:12.651755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:12.651572 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 22:45:13.655536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:13.655507 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:45:13.656696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:13.656681 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:45:15.128022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:15.127992 4298 scope.go:122] "RemoveContainer" containerID="b5a2bf7dde5cd1ab1c0a37b0de412fc12b170a6a100dc458aa8d66a3f2a9e4e2" Nov 03 22:45:16.088527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:16.088494 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:16.128334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:16.128309 4298 scope.go:122] "RemoveContainer" containerID="b5cf9760a8b540d46859dec27af90bcf463f7379f61c65374a2ddfc40fa437b2" Nov 03 22:45:16.128856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:16.128694 4298 scope.go:122] "RemoveContainer" containerID="92c91eb93dd61f8a46d14dfcbb7dde1140f3c889d8c6b147deebd5ac2f1d89e7" Nov 03 22:45:16.241038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:16.241006 4298 generic.go:356] "Generic (PLEG): container finished" podID="22571f32-7943-4e26-96f5-f409490f2969" containerID="65d0a650adfd3c4a13bbb3d33d73382d408bd577c042f1d8c0a42652fe229293" exitCode=60 Nov 03 22:45:16.241194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:16.241076 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" event={"ID":"22571f32-7943-4e26-96f5-f409490f2969","Type":"ContainerDied","Data":"65d0a650adfd3c4a13bbb3d33d73382d408bd577c042f1d8c0a42652fe229293"} Nov 03 22:45:16.241194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:16.241134 4298 scope.go:122] "RemoveContainer" containerID="b5a2bf7dde5cd1ab1c0a37b0de412fc12b170a6a100dc458aa8d66a3f2a9e4e2" Nov 03 22:45:16.241559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:16.241537 4298 scope.go:122] "RemoveContainer" containerID="65d0a650adfd3c4a13bbb3d33d73382d408bd577c042f1d8c0a42652fe229293" Nov 03 22:45:16.241962 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:16.241940 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b_odh-ai-gateway-infra(22571f32-7943-4e26-96f5-f409490f2969)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" podUID="22571f32-7943-4e26-96f5-f409490f2969" Nov 03 22:45:16.262704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:16.262616 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:16.839608 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:16.839581 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:17.128601 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.128512 4298 scope.go:122] "RemoveContainer" containerID="020aef95d21d12842fde0d134dea4565b6ddae6ca60db3c05a4f1a25950e1db0" Nov 03 22:45:17.129117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.128687 4298 scope.go:122] "RemoveContainer" containerID="689d936d6f0d58c966c8f8df944319bc45bb7a44e8ef2a303769fb8e3e1ed3de" Nov 03 22:45:17.129117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.128825 4298 scope.go:122] "RemoveContainer" containerID="c6d9f2797bb366d9a5d73ad03637d994772e68474d815ff2e4d8d3edfbc4402d" Nov 03 22:45:17.252704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.252671 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" event={"ID":"c029cafa-94ab-4b6a-b775-3fee6c1acea6","Type":"ContainerStarted","Data":"971f0c123e66d5d1b7976ca8181ce74528d05817c5e1b2e8ca05ea7367ca8d8a"} Nov 03 22:45:17.257710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.257682 4298 generic.go:356] "Generic (PLEG): container finished" podID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerID="79d754bec4797b7aea674cce0c753dd03707bacccb058e80fa989a93b65dfde1" exitCode=60 Nov 03 22:45:17.257899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.257766 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" event={"ID":"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12","Type":"ContainerDied","Data":"79d754bec4797b7aea674cce0c753dd03707bacccb058e80fa989a93b65dfde1"} Nov 03 22:45:17.257899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.257834 4298 scope.go:122] "RemoveContainer" containerID="92c91eb93dd61f8a46d14dfcbb7dde1140f3c889d8c6b147deebd5ac2f1d89e7" Nov 03 22:45:17.258165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.258145 4298 scope.go:122] "RemoveContainer" containerID="79d754bec4797b7aea674cce0c753dd03707bacccb058e80fa989a93b65dfde1" Nov 03 22:45:17.258471 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:17.258452 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz_odh-ai-gateway-infra(21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" Nov 03 22:45:17.282399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.282361 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:17.312293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.312265 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d"] Nov 03 22:45:17.313010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.312995 4298 cpu_manager.go:414] "containerMap: removing container" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" Nov 03 22:45:17.313082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.313011 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" Nov 03 22:45:17.313158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.313149 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2592c2-0c5e-4e4d-a01e-2fb1e382d3b0" containerName="main" Nov 03 22:45:17.317864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.317842 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.402715 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.375009 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7bcd0b-openshift-default-dockercfg-w6zgv\"" Nov 03 22:45:17.522668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.517151 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-socket\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.522668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.517273 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-certs\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.522668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.517328 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-istio-token\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.522668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.517374 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-credential-socket\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.522668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.517466 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe800041-c967-421b-a414-2a6aebb1d69c-istiod-ca-cert\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.522668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.517498 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-data\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.522668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.517552 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-envoy\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.522668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.517631 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6tb\" (UniqueName: \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-kube-api-access-5k6tb\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.522668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.517844 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe800041-c967-421b-a414-2a6aebb1d69c-istio-podinfo\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.544130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.543367 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:17.554690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.551806 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b"] Nov 03 22:45:17.634221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.631161 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe800041-c967-421b-a414-2a6aebb1d69c-istiod-ca-cert\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.634221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.631238 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-data\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.634221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.631681 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-data\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.634221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.631832 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-envoy\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.637664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.635606 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6tb\" (UniqueName: \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-kube-api-access-5k6tb\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.656770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.635555 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe800041-c967-421b-a414-2a6aebb1d69c-istiod-ca-cert\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.656770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.638408 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe800041-c967-421b-a414-2a6aebb1d69c-istio-podinfo\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.656770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.646748 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-socket\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.656770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.646861 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-certs\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.656770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.646926 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-istio-token\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.656770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.646984 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-credential-socket\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.683247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.680130 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-credential-socket\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.683247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.680858 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-socket\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.683247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.681126 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-certs\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.683247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.681455 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-envoy\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.683984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.683662 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d"] Nov 03 22:45:17.685989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.685962 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe800041-c967-421b-a414-2a6aebb1d69c-istio-podinfo\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.695255 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:17.695164 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:45:17.699106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.699061 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6tb\" (UniqueName: \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-kube-api-access-5k6tb\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.743499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.743437 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-istio-token\") pod \"e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.815731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.815704 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" Nov 03 22:45:17.843522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.843491 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:17.943632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.943457 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:17.954852 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.953837 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22571f32-7943-4e26-96f5-f409490f2969-kube-api-access-ghxcc\" (UniqueName: \"kubernetes.io/projected/22571f32-7943-4e26-96f5-f409490f2969-kube-api-access-ghxcc\") pod \"22571f32-7943-4e26-96f5-f409490f2969\" (UID: \"22571f32-7943-4e26-96f5-f409490f2969\") " Nov 03 22:45:17.954852 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.954076 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22571f32-7943-4e26-96f5-f409490f2969-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22571f32-7943-4e26-96f5-f409490f2969-maas-api-service-ca\") pod \"22571f32-7943-4e26-96f5-f409490f2969\" (UID: \"22571f32-7943-4e26-96f5-f409490f2969\") " Nov 03 22:45:17.955566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.955241 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22571f32-7943-4e26-96f5-f409490f2969-maas-api-service-ca" pod "22571f32-7943-4e26-96f5-f409490f2969" (UID: "22571f32-7943-4e26-96f5-f409490f2969"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:17.958748 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:17.958707 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22571f32-7943-4e26-96f5-f409490f2969-kube-api-access-ghxcc" pod "22571f32-7943-4e26-96f5-f409490f2969" (UID: "22571f32-7943-4e26-96f5-f409490f2969"). InnerVolumeSpecName "kube-api-access-ghxcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:18.056373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.056339 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22571f32-7943-4e26-96f5-f409490f2969-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:18.056373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.056370 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghxcc\" (UniqueName: \"kubernetes.io/projected/22571f32-7943-4e26-96f5-f409490f2969-kube-api-access-ghxcc\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:18.128443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.128400 4298 scope.go:122] "RemoveContainer" containerID="bc9440ee56644aebb4dc7e46bd98585c2e2c95169be7ed3e6d0a47ad6c2b493f" Nov 03 22:45:18.128745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.128723 4298 scope.go:122] "RemoveContainer" containerID="ed0c91736c511c3d345830b440ff0aab7c7fee783a912afcd92c7849a37b469b" Nov 03 22:45:18.129132 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.128812 4298 scope.go:122] "RemoveContainer" containerID="31cf87aebe6e0fa1d5bfe6bd5ecafbe8f5a22afbec4d57754c334901638b4747" Nov 03 22:45:18.129132 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.128849 4298 scope.go:122] "RemoveContainer" containerID="5e231792f7eec04ca45d8193fe9ca71b036f6729ab844dab7021e79646c2085a" Nov 03 22:45:18.273698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.273638 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" event={"ID":"03b37944-6808-42f4-857f-57ea8635a2cd","Type":"ContainerStarted","Data":"f237554dd690cefb507c143a4115f1756c7b72780739dbf76d6ecb98d70aee98"} Nov 03 22:45:18.275167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.275141 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" event={"ID":"22571f32-7943-4e26-96f5-f409490f2969","Type":"ContainerDied","Data":"8189f375f9d5a3a906d3df3b68d90b75615bb3321c10bcd0264ed9881ec690c7"} Nov 03 22:45:18.275167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.275155 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b" Nov 03 22:45:18.275384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.275196 4298 scope.go:122] "RemoveContainer" containerID="65d0a650adfd3c4a13bbb3d33d73382d408bd577c042f1d8c0a42652fe229293" Nov 03 22:45:18.278966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.278938 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" event={"ID":"157ee087-43db-4f2f-879d-a6e477ddf041","Type":"ContainerStarted","Data":"d0afc4dff5f3559df6a4033df60ffe056d275e2f85a60adc814160c9ab8fab1f"} Nov 03 22:45:18.281156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.281134 4298 generic.go:356] "Generic (PLEG): container finished" podID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerID="971f0c123e66d5d1b7976ca8181ce74528d05817c5e1b2e8ca05ea7367ca8d8a" exitCode=60 Nov 03 22:45:18.281261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.281203 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" event={"ID":"c029cafa-94ab-4b6a-b775-3fee6c1acea6","Type":"ContainerDied","Data":"971f0c123e66d5d1b7976ca8181ce74528d05817c5e1b2e8ca05ea7367ca8d8a"} Nov 03 22:45:18.281522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.281495 4298 scope.go:122] "RemoveContainer" containerID="971f0c123e66d5d1b7976ca8181ce74528d05817c5e1b2e8ca05ea7367ca8d8a" Nov 03 22:45:18.281793 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:18.281774 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6_odh-ai-gateway-infra(c029cafa-94ab-4b6a-b775-3fee6c1acea6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" Nov 03 22:45:18.283184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.283165 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" event={"ID":"1534c655-737d-499d-8fcf-00e85c96854c","Type":"ContainerStarted","Data":"883835fd1d358d1cd085034d988760978fa9a07ad517f85071b4e14dfdb3a399"} Nov 03 22:45:18.286989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.286974 4298 scope.go:122] "RemoveContainer" containerID="b5cf9760a8b540d46859dec27af90bcf463f7379f61c65374a2ddfc40fa437b2" Nov 03 22:45:18.306373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.306347 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d"] Nov 03 22:45:18.338071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.338032 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:18.338196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.338128 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:18.338196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.338174 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:18.475460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.475379 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6"] Nov 03 22:45:18.509057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.509019 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:18.638617 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.638587 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz"] Nov 03 22:45:18.700702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.695088 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:18.777664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.775805 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b"] Nov 03 22:45:18.796189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.796160 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41bb2b-29895765-z8w4b"] Nov 03 22:45:18.935588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.935560 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" Nov 03 22:45:18.991150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:18.991056 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:19.083777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.083744 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-kube-api-access-r5cl5\" (UniqueName: \"kubernetes.io/projected/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-kube-api-access-r5cl5\") pod \"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12\" (UID: \"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12\") " Nov 03 22:45:19.083965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.083903 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-maas-api-service-ca\") pod \"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12\" (UID: \"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12\") " Nov 03 22:45:19.084331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.084281 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-maas-api-service-ca" pod "21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" (UID: "21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:19.084577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.084562 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:19.085957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.085929 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-kube-api-access-r5cl5" pod "21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" (UID: "21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12"). InnerVolumeSpecName "kube-api-access-r5cl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:19.134382 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.134351 4298 scope.go:122] "RemoveContainer" containerID="1d838ccdc93e162f1102f4a17880babb5cd0e8766abbd58b5a2e077bb0f62feb" Nov 03 22:45:19.144387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.144341 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22571f32-7943-4e26-96f5-f409490f2969" path="/var/lib/kubelet/pods/22571f32-7943-4e26-96f5-f409490f2969/volumes" Nov 03 22:45:19.185660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.185623 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5cl5\" (UniqueName: \"kubernetes.io/projected/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12-kube-api-access-r5cl5\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:19.294127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.294050 4298 generic.go:356] "Generic (PLEG): container finished" podID="1534c655-737d-499d-8fcf-00e85c96854c" containerID="883835fd1d358d1cd085034d988760978fa9a07ad517f85071b4e14dfdb3a399" exitCode=60 Nov 03 22:45:19.294293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.294130 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" event={"ID":"1534c655-737d-499d-8fcf-00e85c96854c","Type":"ContainerDied","Data":"883835fd1d358d1cd085034d988760978fa9a07ad517f85071b4e14dfdb3a399"} Nov 03 22:45:19.294293 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.294211 4298 scope.go:122] "RemoveContainer" containerID="020aef95d21d12842fde0d134dea4565b6ddae6ca60db3c05a4f1a25950e1db0" Nov 03 22:45:19.294599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.294585 4298 scope.go:122] "RemoveContainer" containerID="883835fd1d358d1cd085034d988760978fa9a07ad517f85071b4e14dfdb3a399" Nov 03 22:45:19.294940 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:19.294915 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd_odh-ai-gateway-infra(1534c655-737d-499d-8fcf-00e85c96854c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" podUID="1534c655-737d-499d-8fcf-00e85c96854c" Nov 03 22:45:19.296591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.296568 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" event={"ID":"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17","Type":"ContainerStarted","Data":"80de714e88e3c4f0623806b505f576f7c2cbb401193ab1690e4fe3b7d2379c1b"} Nov 03 22:45:19.298497 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.298477 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" event={"ID":"ecc2af8c-c3b6-4bde-ac6f-398533e20acf","Type":"ContainerStarted","Data":"9d6489c1fda8636a683c1466b558c4350edd65a972c4cdc2fc772cebea832084"} Nov 03 22:45:19.300555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.300532 4298 generic.go:356] "Generic (PLEG): container finished" podID="03b37944-6808-42f4-857f-57ea8635a2cd" containerID="f237554dd690cefb507c143a4115f1756c7b72780739dbf76d6ecb98d70aee98" exitCode=60 Nov 03 22:45:19.300638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.300600 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" event={"ID":"03b37944-6808-42f4-857f-57ea8635a2cd","Type":"ContainerDied","Data":"f237554dd690cefb507c143a4115f1756c7b72780739dbf76d6ecb98d70aee98"} Nov 03 22:45:19.300993 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.300968 4298 scope.go:122] "RemoveContainer" containerID="f237554dd690cefb507c143a4115f1756c7b72780739dbf76d6ecb98d70aee98" Nov 03 22:45:19.301233 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:19.301213 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895765-vbght_odh-ai-gateway-infra(03b37944-6808-42f4-857f-57ea8635a2cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" Nov 03 22:45:19.302768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.302748 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" event={"ID":"f3056f50-50c4-4804-af55-0e9e4f06f583","Type":"ContainerStarted","Data":"c6253c25b5215a0a5b36c5f4519986f99cafd9e2dbc87f7d8c1c8f46b2299840"} Nov 03 22:45:19.304591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.304567 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" event={"ID":"fe800041-c967-421b-a414-2a6aebb1d69c","Type":"ContainerStarted","Data":"d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d"} Nov 03 22:45:19.304740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.304613 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" event={"ID":"fe800041-c967-421b-a414-2a6aebb1d69c","Type":"ContainerStarted","Data":"6c9ed299446d69745db5390e9d3737e0f465f528da1504989bb475a492c677fd"} Nov 03 22:45:19.307097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.307080 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" Nov 03 22:45:19.307216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.307105 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz" event={"ID":"21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12","Type":"ContainerDied","Data":"81639ad591033d5bfabe59af3e4fa23236bf45bd6294c047761b9a2f04f72ad5"} Nov 03 22:45:19.309740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.309376 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" event={"ID":"50000cc4-9bd6-4cfb-a85f-15b0ae191071","Type":"ContainerStarted","Data":"36ecc8902915c49212593186367ed1c3de167ca42085b757bd760e1ada1ce8cc"} Nov 03 22:45:19.311493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.311474 4298 generic.go:356] "Generic (PLEG): container finished" podID="157ee087-43db-4f2f-879d-a6e477ddf041" containerID="d0afc4dff5f3559df6a4033df60ffe056d275e2f85a60adc814160c9ab8fab1f" exitCode=60 Nov 03 22:45:19.311588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.311541 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" event={"ID":"157ee087-43db-4f2f-879d-a6e477ddf041","Type":"ContainerDied","Data":"d0afc4dff5f3559df6a4033df60ffe056d275e2f85a60adc814160c9ab8fab1f"} Nov 03 22:45:19.311866 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.311848 4298 scope.go:122] "RemoveContainer" containerID="689d936d6f0d58c966c8f8df944319bc45bb7a44e8ef2a303769fb8e3e1ed3de" Nov 03 22:45:19.311938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.311907 4298 scope.go:122] "RemoveContainer" containerID="d0afc4dff5f3559df6a4033df60ffe056d275e2f85a60adc814160c9ab8fab1f" Nov 03 22:45:19.312131 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:19.312114 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr_odh-ai-gateway-infra(157ee087-43db-4f2f-879d-a6e477ddf041)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" Nov 03 22:45:19.329542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.329519 4298 scope.go:122] "RemoveContainer" containerID="79d754bec4797b7aea674cce0c753dd03707bacccb058e80fa989a93b65dfde1" Nov 03 22:45:19.383611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.383573 4298 scope.go:122] "RemoveContainer" containerID="c6d9f2797bb366d9a5d73ad03637d994772e68474d815ff2e4d8d3edfbc4402d" Nov 03 22:45:19.402927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.402897 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght"] Nov 03 22:45:19.616772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.616747 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" Nov 03 22:45:19.617132 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617114 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm"] Nov 03 22:45:19.617800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617785 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerName="cleanup" Nov 03 22:45:19.617838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617801 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerName="cleanup" Nov 03 22:45:19.617838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617812 4298 cpu_manager.go:414] "containerMap: removing container" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerName="cleanup" Nov 03 22:45:19.617838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617817 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerName="cleanup" Nov 03 22:45:19.617838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617823 4298 cpu_manager.go:414] "containerMap: removing container" podUID="22571f32-7943-4e26-96f5-f409490f2969" containerName="cleanup" Nov 03 22:45:19.617838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617827 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22571f32-7943-4e26-96f5-f409490f2969" containerName="cleanup" Nov 03 22:45:19.618001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617844 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerName="cleanup" Nov 03 22:45:19.618001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617848 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerName="cleanup" Nov 03 22:45:19.618001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617857 4298 cpu_manager.go:414] "containerMap: removing container" podUID="22571f32-7943-4e26-96f5-f409490f2969" containerName="cleanup" Nov 03 22:45:19.618001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617861 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22571f32-7943-4e26-96f5-f409490f2969" containerName="cleanup" Nov 03 22:45:19.618001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617872 4298 cpu_manager.go:414] "containerMap: removing container" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerName="cleanup" Nov 03 22:45:19.618001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617877 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerName="cleanup" Nov 03 22:45:19.618001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617884 4298 cpu_manager.go:414] "containerMap: removing container" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerName="cleanup" Nov 03 22:45:19.618001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617888 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerName="cleanup" Nov 03 22:45:19.618001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617902 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerName="cleanup" Nov 03 22:45:19.618001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.617907 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerName="cleanup" Nov 03 22:45:19.618470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.618016 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerName="cleanup" Nov 03 22:45:19.618470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.618026 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="22571f32-7943-4e26-96f5-f409490f2969" containerName="cleanup" Nov 03 22:45:19.618470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.618035 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerName="cleanup" Nov 03 22:45:19.618470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.618043 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerName="cleanup" Nov 03 22:45:19.618470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.618048 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="22571f32-7943-4e26-96f5-f409490f2969" containerName="cleanup" Nov 03 22:45:19.618470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.618054 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerName="cleanup" Nov 03 22:45:19.621604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.621584 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.630954 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.630929 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk"] Nov 03 22:45:19.631956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.631935 4298 cpu_manager.go:414] "containerMap: removing container" podUID="22571f32-7943-4e26-96f5-f409490f2969" containerName="cleanup" Nov 03 22:45:19.632059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.631957 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22571f32-7943-4e26-96f5-f409490f2969" containerName="cleanup" Nov 03 22:45:19.632197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.632185 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" containerName="cleanup" Nov 03 22:45:19.632253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.632215 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" containerName="cleanup" Nov 03 22:45:19.632253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.632231 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="22571f32-7943-4e26-96f5-f409490f2969" containerName="cleanup" Nov 03 22:45:19.636421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.636401 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.640677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.638018 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7"] Nov 03 22:45:19.643372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.643350 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.651511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.651487 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr"] Nov 03 22:45:19.679936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.679903 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kf4rv\"" Nov 03 22:45:19.683689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.683654 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm"] Nov 03 22:45:19.697678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.697476 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk"] Nov 03 22:45:19.738533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.738499 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7"] Nov 03 22:45:19.798602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.798567 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c029cafa-94ab-4b6a-b775-3fee6c1acea6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c029cafa-94ab-4b6a-b775-3fee6c1acea6-maas-api-service-ca\") pod \"c029cafa-94ab-4b6a-b775-3fee6c1acea6\" (UID: \"c029cafa-94ab-4b6a-b775-3fee6c1acea6\") " Nov 03 22:45:19.798833 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.798767 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c029cafa-94ab-4b6a-b775-3fee6c1acea6-kube-api-access-8qs8k\" (UniqueName: \"kubernetes.io/projected/c029cafa-94ab-4b6a-b775-3fee6c1acea6-kube-api-access-8qs8k\") pod \"c029cafa-94ab-4b6a-b775-3fee6c1acea6\" (UID: \"c029cafa-94ab-4b6a-b775-3fee6c1acea6\") " Nov 03 22:45:19.798984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.798956 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c029cafa-94ab-4b6a-b775-3fee6c1acea6-maas-api-service-ca" pod "c029cafa-94ab-4b6a-b775-3fee6c1acea6" (UID: "c029cafa-94ab-4b6a-b775-3fee6c1acea6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:19.798984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.798969 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.799103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799029 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.799103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799050 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b506bf8c-2001-4044-9e2e-b18b427cb33b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.799103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799097 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c82nt\" (UniqueName: \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-kube-api-access-c82nt\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.799285 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799221 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.799285 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799260 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.799285 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799280 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.799466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799385 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.799466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799415 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.799466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799439 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.799625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799487 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.799625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799589 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.799766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799651 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwr82\" (UniqueName: \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-kube-api-access-hwr82\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.799766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799720 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.799766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799760 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.799927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799795 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.799927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799829 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb5598d-15f2-4625-ad51-c65350e4a159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.799927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799890 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.799927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799915 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.800198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.799941 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.800198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.800013 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.800198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.800079 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.800198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.800106 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb5598d-15f2-4625-ad51-c65350e4a159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.800198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.800123 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.800198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.800143 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.800535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.800215 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7qd\" (UniqueName: \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-kube-api-access-kr7qd\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.800535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.800263 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.800535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.800393 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c029cafa-94ab-4b6a-b775-3fee6c1acea6-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:19.801088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.801060 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c029cafa-94ab-4b6a-b775-3fee6c1acea6-kube-api-access-8qs8k" pod "c029cafa-94ab-4b6a-b775-3fee6c1acea6" (UID: "c029cafa-94ab-4b6a-b775-3fee6c1acea6"). InnerVolumeSpecName "kube-api-access-8qs8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:19.844225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.844197 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:19.850056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.850027 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:19.867881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.867793 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz"] Nov 03 22:45:19.872691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.872669 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-fac3b6-29895765-q46wz"] Nov 03 22:45:19.887550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.887524 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd"] Nov 03 22:45:19.901317 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901292 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb5598d-15f2-4625-ad51-c65350e4a159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.901446 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901326 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.901446 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901351 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.901446 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901391 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7qd\" (UniqueName: \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-kube-api-access-kr7qd\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.901620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901449 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.901620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901505 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.901620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901534 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.901620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901559 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b506bf8c-2001-4044-9e2e-b18b427cb33b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.901868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901624 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c82nt\" (UniqueName: \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-kube-api-access-c82nt\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.901868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901695 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.901868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901741 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.901868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901767 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.901868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901831 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.901868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901856 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.902193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901883 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.902193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901908 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.902193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.901948 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.902193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902013 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwr82\" (UniqueName: \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-kube-api-access-hwr82\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.902193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902034 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.902193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902057 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.902193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902104 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.902193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902138 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.902193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902168 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb5598d-15f2-4625-ad51-c65350e4a159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.902193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902190 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.902753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902196 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb5598d-15f2-4625-ad51-c65350e4a159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.902753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902248 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.902753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902290 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.902753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902332 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.902753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902338 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.902753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902375 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.902753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902403 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.902753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902504 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qs8k\" (UniqueName: \"kubernetes.io/projected/c029cafa-94ab-4b6a-b775-3fee6c1acea6-kube-api-access-8qs8k\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:19.902753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902596 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.903231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.902899 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.904231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.903549 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.904231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.903858 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.904231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.903884 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.904231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.904110 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.904535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.904340 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.904535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.904495 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b506bf8c-2001-4044-9e2e-b18b427cb33b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.904663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.904595 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.904897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.904869 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.905121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.905092 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.905468 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.905442 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.905677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.905618 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.905797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.905749 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.906369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.906346 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.906853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.906837 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.907302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.907287 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb5598d-15f2-4625-ad51-c65350e4a159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.959314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.959282 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c82nt\" (UniqueName: \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-kube-api-access-c82nt\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.959582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.959351 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.959700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.959560 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7qd\" (UniqueName: \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-kube-api-access-kr7qd\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.959781 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.959757 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ngvlk\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:19.960919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.960898 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sn4pm\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:19.961067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.961044 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwr82\" (UniqueName: \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-kube-api-access-hwr82\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zkjl7\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:19.961842 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:19.961826 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:20.239750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.239715 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:20.253001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.252894 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:20.346604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.346368 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" podStartSLOduration=3.3463488789999998 podStartE2EDuration="3.346348879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:20.326212941 +0000 UTC m=+2277.306946632" watchObservedRunningTime="2026-11-03 22:45:20.346348879 +0000 UTC m=+2277.327082535" Nov 03 22:45:20.361552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.361068 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:20.361552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.361153 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:20.361552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.361197 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:20.363098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.362892 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7"] Nov 03 22:45:20.365389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.365347 4298 generic.go:356] "Generic (PLEG): container finished" podID="81f863b3-9457-4888-b046-3cfd408e504a" containerID="31a7e71753b3794a9f81ff3c225c9de3dac22b64d5af873786186f648846e160" exitCode=60 Nov 03 22:45:20.365511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.365483 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" event={"ID":"81f863b3-9457-4888-b046-3cfd408e504a","Type":"ContainerDied","Data":"31a7e71753b3794a9f81ff3c225c9de3dac22b64d5af873786186f648846e160"} Nov 03 22:45:20.365573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.365557 4298 scope.go:122] "RemoveContainer" containerID="1d838ccdc93e162f1102f4a17880babb5cd0e8766abbd58b5a2e077bb0f62feb" Nov 03 22:45:20.366281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.366106 4298 scope.go:122] "RemoveContainer" containerID="31a7e71753b3794a9f81ff3c225c9de3dac22b64d5af873786186f648846e160" Nov 03 22:45:20.371062 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:20.367841 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr_odh-ai-gateway-infra(81f863b3-9457-4888-b046-3cfd408e504a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" podUID="81f863b3-9457-4888-b046-3cfd408e504a" Nov 03 22:45:20.375693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.375109 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" event={"ID":"c029cafa-94ab-4b6a-b775-3fee6c1acea6","Type":"ContainerDied","Data":"d150b45413534c29dfe15b12fdaab333b0bec9086468d51ffdee4807d4f0a779"} Nov 03 22:45:20.375693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.375367 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6" Nov 03 22:45:20.381705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.381681 4298 generic.go:356] "Generic (PLEG): container finished" podID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerID="80de714e88e3c4f0623806b505f576f7c2cbb401193ab1690e4fe3b7d2379c1b" exitCode=60 Nov 03 22:45:20.381954 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.381932 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" event={"ID":"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17","Type":"ContainerDied","Data":"80de714e88e3c4f0623806b505f576f7c2cbb401193ab1690e4fe3b7d2379c1b"} Nov 03 22:45:20.382636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.382616 4298 scope.go:122] "RemoveContainer" containerID="80de714e88e3c4f0623806b505f576f7c2cbb401193ab1690e4fe3b7d2379c1b" Nov 03 22:45:20.383288 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:20.383265 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6_odh-ai-gateway-infra(9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" Nov 03 22:45:20.386049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.385992 4298 generic.go:356] "Generic (PLEG): container finished" podID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerID="9d6489c1fda8636a683c1466b558c4350edd65a972c4cdc2fc772cebea832084" exitCode=60 Nov 03 22:45:20.386276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.386257 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" event={"ID":"ecc2af8c-c3b6-4bde-ac6f-398533e20acf","Type":"ContainerDied","Data":"9d6489c1fda8636a683c1466b558c4350edd65a972c4cdc2fc772cebea832084"} Nov 03 22:45:20.386913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.386893 4298 scope.go:122] "RemoveContainer" containerID="9d6489c1fda8636a683c1466b558c4350edd65a972c4cdc2fc772cebea832084" Nov 03 22:45:20.387492 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:20.387467 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6_odh-ai-gateway-infra(ecc2af8c-c3b6-4bde-ac6f-398533e20acf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" Nov 03 22:45:20.393777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.393753 4298 generic.go:356] "Generic (PLEG): container finished" podID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerID="c6253c25b5215a0a5b36c5f4519986f99cafd9e2dbc87f7d8c1c8f46b2299840" exitCode=60 Nov 03 22:45:20.393996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.393976 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" event={"ID":"f3056f50-50c4-4804-af55-0e9e4f06f583","Type":"ContainerDied","Data":"c6253c25b5215a0a5b36c5f4519986f99cafd9e2dbc87f7d8c1c8f46b2299840"} Nov 03 22:45:20.394976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.394956 4298 scope.go:122] "RemoveContainer" containerID="c6253c25b5215a0a5b36c5f4519986f99cafd9e2dbc87f7d8c1c8f46b2299840" Nov 03 22:45:20.396234 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:20.396209 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp_odh-ai-gateway-infra(f3056f50-50c4-4804-af55-0e9e4f06f583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" Nov 03 22:45:20.406348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.405277 4298 generic.go:356] "Generic (PLEG): container finished" podID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerID="36ecc8902915c49212593186367ed1c3de167ca42085b757bd760e1ada1ce8cc" exitCode=60 Nov 03 22:45:20.406348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.405725 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" event={"ID":"50000cc4-9bd6-4cfb-a85f-15b0ae191071","Type":"ContainerDied","Data":"36ecc8902915c49212593186367ed1c3de167ca42085b757bd760e1ada1ce8cc"} Nov 03 22:45:20.406780 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.406763 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:20.409681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.409495 4298 scope.go:122] "RemoveContainer" containerID="36ecc8902915c49212593186367ed1c3de167ca42085b757bd760e1ada1ce8cc" Nov 03 22:45:20.410036 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:20.410010 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn_odh-ai-gateway-infra(50000cc4-9bd6-4cfb-a85f-15b0ae191071)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" Nov 03 22:45:20.412583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.412560 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:45:20.484188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.484155 4298 scope.go:122] "RemoveContainer" containerID="971f0c123e66d5d1b7976ca8181ce74528d05817c5e1b2e8ca05ea7367ca8d8a" Nov 03 22:45:20.522515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.522481 4298 scope.go:122] "RemoveContainer" containerID="ed0c91736c511c3d345830b440ff0aab7c7fee783a912afcd92c7849a37b469b" Nov 03 22:45:20.552334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.552246 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn"] Nov 03 22:45:20.583865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.583046 4298 scope.go:122] "RemoveContainer" containerID="31cf87aebe6e0fa1d5bfe6bd5ecafbe8f5a22afbec4d57754c334901638b4747" Nov 03 22:45:20.609890 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.609782 4298 scope.go:122] "RemoveContainer" containerID="bc9440ee56644aebb4dc7e46bd98585c2e2c95169be7ed3e6d0a47ad6c2b493f" Nov 03 22:45:20.668885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.650555 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm"] Nov 03 22:45:20.668885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.653597 4298 scope.go:122] "RemoveContainer" containerID="5e231792f7eec04ca45d8193fe9ca71b036f6729ab844dab7021e79646c2085a" Nov 03 22:45:20.668885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.661421 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:20.668885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.661501 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:20.668885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.661542 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:20.668885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.666833 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:20.668885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.666904 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:20.668885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.666943 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:20.669402 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.669316 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk"] Nov 03 22:45:20.677885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.677804 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp"] Nov 03 22:45:20.707394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.689775 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" Nov 03 22:45:20.771387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.771275 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6"] Nov 03 22:45:20.780252 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.779104 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" Nov 03 22:45:20.783843 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.783823 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" Nov 03 22:45:20.799555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.799530 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:20.834251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.833079 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157ee087-43db-4f2f-879d-a6e477ddf041-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/157ee087-43db-4f2f-879d-a6e477ddf041-maas-api-service-ca\") pod \"157ee087-43db-4f2f-879d-a6e477ddf041\" (UID: \"157ee087-43db-4f2f-879d-a6e477ddf041\") " Nov 03 22:45:20.834251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.833221 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157ee087-43db-4f2f-879d-a6e477ddf041-kube-api-access-t8jhx\" (UniqueName: \"kubernetes.io/projected/157ee087-43db-4f2f-879d-a6e477ddf041-kube-api-access-t8jhx\") pod \"157ee087-43db-4f2f-879d-a6e477ddf041\" (UID: \"157ee087-43db-4f2f-879d-a6e477ddf041\") " Nov 03 22:45:20.836072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.836033 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157ee087-43db-4f2f-879d-a6e477ddf041-maas-api-service-ca" pod "157ee087-43db-4f2f-879d-a6e477ddf041" (UID: "157ee087-43db-4f2f-879d-a6e477ddf041"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:20.836324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.836302 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157ee087-43db-4f2f-879d-a6e477ddf041-kube-api-access-t8jhx" pod "157ee087-43db-4f2f-879d-a6e477ddf041" (UID: "157ee087-43db-4f2f-879d-a6e477ddf041"). InnerVolumeSpecName "kube-api-access-t8jhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:20.934733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.934631 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1534c655-737d-499d-8fcf-00e85c96854c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1534c655-737d-499d-8fcf-00e85c96854c-maas-api-service-ca\") pod \"1534c655-737d-499d-8fcf-00e85c96854c\" (UID: \"1534c655-737d-499d-8fcf-00e85c96854c\") " Nov 03 22:45:20.935080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.935054 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b37944-6808-42f4-857f-57ea8635a2cd-kube-api-access-9948q\" (UniqueName: \"kubernetes.io/projected/03b37944-6808-42f4-857f-57ea8635a2cd-kube-api-access-9948q\") pod \"03b37944-6808-42f4-857f-57ea8635a2cd\" (UID: \"03b37944-6808-42f4-857f-57ea8635a2cd\") " Nov 03 22:45:20.935283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.935267 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1534c655-737d-499d-8fcf-00e85c96854c-kube-api-access-whsdj\" (UniqueName: \"kubernetes.io/projected/1534c655-737d-499d-8fcf-00e85c96854c-kube-api-access-whsdj\") pod \"1534c655-737d-499d-8fcf-00e85c96854c\" (UID: \"1534c655-737d-499d-8fcf-00e85c96854c\") " Nov 03 22:45:20.935410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.935394 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b37944-6808-42f4-857f-57ea8635a2cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b37944-6808-42f4-857f-57ea8635a2cd-maas-api-service-ca\") pod \"03b37944-6808-42f4-857f-57ea8635a2cd\" (UID: \"03b37944-6808-42f4-857f-57ea8635a2cd\") " Nov 03 22:45:20.936187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.936162 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/157ee087-43db-4f2f-879d-a6e477ddf041-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:20.936336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.936322 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8jhx\" (UniqueName: \"kubernetes.io/projected/157ee087-43db-4f2f-879d-a6e477ddf041-kube-api-access-t8jhx\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:20.936659 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.936620 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b37944-6808-42f4-857f-57ea8635a2cd-maas-api-service-ca" pod "03b37944-6808-42f4-857f-57ea8635a2cd" (UID: "03b37944-6808-42f4-857f-57ea8635a2cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:20.936971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.936736 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1534c655-737d-499d-8fcf-00e85c96854c-maas-api-service-ca" pod "1534c655-737d-499d-8fcf-00e85c96854c" (UID: "1534c655-737d-499d-8fcf-00e85c96854c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:20.939112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.939086 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1534c655-737d-499d-8fcf-00e85c96854c-kube-api-access-whsdj" pod "1534c655-737d-499d-8fcf-00e85c96854c" (UID: "1534c655-737d-499d-8fcf-00e85c96854c"). InnerVolumeSpecName "kube-api-access-whsdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:20.939222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:20.939127 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b37944-6808-42f4-857f-57ea8635a2cd-kube-api-access-9948q" pod "03b37944-6808-42f4-857f-57ea8635a2cd" (UID: "03b37944-6808-42f4-857f-57ea8635a2cd"). InnerVolumeSpecName "kube-api-access-9948q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:21.037618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.037582 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1534c655-737d-499d-8fcf-00e85c96854c-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:21.037618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.037617 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9948q\" (UniqueName: \"kubernetes.io/projected/03b37944-6808-42f4-857f-57ea8635a2cd-kube-api-access-9948q\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:21.037895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.037633 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whsdj\" (UniqueName: \"kubernetes.io/projected/1534c655-737d-499d-8fcf-00e85c96854c-kube-api-access-whsdj\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:21.037895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.037661 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b37944-6808-42f4-857f-57ea8635a2cd-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:21.115377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.115345 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6"] Nov 03 22:45:21.199269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.199230 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12" path="/var/lib/kubelet/pods/21c34f8d-7c95-4d4d-bfe9-0f5365ef0d12/volumes" Nov 03 22:45:21.203883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.203865 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6"] Nov 03 22:45:21.206131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.206063 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f7e6db-29895765-vblk6"] Nov 03 22:45:21.501467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.501433 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" event={"ID":"b506bf8c-2001-4044-9e2e-b18b427cb33b","Type":"ContainerStarted","Data":"c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689"} Nov 03 22:45:21.502958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.502196 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" event={"ID":"b506bf8c-2001-4044-9e2e-b18b427cb33b","Type":"ContainerStarted","Data":"5ac1008451fbbe3986d623af96e860ce8583e1e86c719a9f553f61e0aff402c3"} Nov 03 22:45:21.508023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.507998 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" event={"ID":"157ee087-43db-4f2f-879d-a6e477ddf041","Type":"ContainerDied","Data":"a6393e97ffba37711308582b034c7e4ef5c90864bdf63c434de605b020edc418"} Nov 03 22:45:21.508249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.508205 4298 scope.go:122] "RemoveContainer" containerID="d0afc4dff5f3559df6a4033df60ffe056d275e2f85a60adc814160c9ab8fab1f" Nov 03 22:45:21.508598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.508574 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr" Nov 03 22:45:21.551483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.551410 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" event={"ID":"1534c655-737d-499d-8fcf-00e85c96854c","Type":"ContainerDied","Data":"77b7e55fa0781cb3be5e2031af8011dd9612f1abf525dca5b497b89da4389ec0"} Nov 03 22:45:21.551926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.551897 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd" Nov 03 22:45:21.573470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.573437 4298 scope.go:122] "RemoveContainer" containerID="883835fd1d358d1cd085034d988760978fa9a07ad517f85071b4e14dfdb3a399" Nov 03 22:45:21.576983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.576930 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" event={"ID":"edb5598d-15f2-4625-ad51-c65350e4a159","Type":"ContainerStarted","Data":"d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89"} Nov 03 22:45:21.577114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.576997 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" event={"ID":"edb5598d-15f2-4625-ad51-c65350e4a159","Type":"ContainerStarted","Data":"9f4c5148afbee13e303c19bcb4e2f5047cd03ca4c9701d42921e13d6835205d4"} Nov 03 22:45:21.610041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.609997 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" event={"ID":"b8bd390f-7aef-4788-84cb-ae1ae867c6d1","Type":"ContainerStarted","Data":"b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a"} Nov 03 22:45:21.610262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.610245 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" event={"ID":"b8bd390f-7aef-4788-84cb-ae1ae867c6d1","Type":"ContainerStarted","Data":"52e33a4932c793e335685198af10758b1b74fa5a90b947ea8389841de36e0e28"} Nov 03 22:45:21.613093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.613071 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" event={"ID":"03b37944-6808-42f4-857f-57ea8635a2cd","Type":"ContainerDied","Data":"c785d0aa0ff0c3d6466d164c83fd405c954295e858f561dc337e55658650c6cd"} Nov 03 22:45:21.613336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.613322 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght" Nov 03 22:45:21.786287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.783005 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr"] Nov 03 22:45:21.874042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:21.817811 4298 scope.go:122] "RemoveContainer" containerID="f237554dd690cefb507c143a4115f1756c7b72780739dbf76d6ecb98d70aee98" Nov 03 22:45:22.016584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.016510 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:22.018943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.016978 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:45:22.019676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.017038 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:45:22.261121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.243383 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:22.299264 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.299171 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:22.346715 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.277099 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 03 22:45:22.347012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.346978 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 03 22:45:22.453669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.450175 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 03 22:45:22.461766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.461723 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 03 22:45:22.488946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.483531 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:22.822240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.822171 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" Nov 03 22:45:22.822823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.822451 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" event={"ID":"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17","Type":"ContainerDied","Data":"ba86baabbac423d6c70ce421b2f48bfffe0103be51c36275f65182e2c13168ce"} Nov 03 22:45:22.823046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.823030 4298 scope.go:122] "RemoveContainer" containerID="80de714e88e3c4f0623806b505f576f7c2cbb401193ab1690e4fe3b7d2379c1b" Nov 03 22:45:22.916671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.909539 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-maas-api-service-ca\") pod \"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17\" (UID: \"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17\") " Nov 03 22:45:22.916671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.909920 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-kube-api-access-cjg2z\" (UniqueName: \"kubernetes.io/projected/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-kube-api-access-cjg2z\") pod \"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17\" (UID: \"9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17\") " Nov 03 22:45:22.920962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.920938 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-maas-api-service-ca" pod "9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" (UID: "9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:22.923384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.923362 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" event={"ID":"50000cc4-9bd6-4cfb-a85f-15b0ae191071","Type":"ContainerDied","Data":"c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c"} Nov 03 22:45:22.923521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:22.923510 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c" Nov 03 22:45:23.002896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.002864 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" Nov 03 22:45:23.014802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.013085 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:45:23.014802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.013138 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:45:23.014802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.013346 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.020405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.020178 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-kube-api-access-cjg2z" pod "9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" (UID: "9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17"). InnerVolumeSpecName "kube-api-access-cjg2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:23.108557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.108527 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" Nov 03 22:45:23.114223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.114195 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50000cc4-9bd6-4cfb-a85f-15b0ae191071-kube-api-access-ml2fk\" (UniqueName: \"kubernetes.io/projected/50000cc4-9bd6-4cfb-a85f-15b0ae191071-kube-api-access-ml2fk\") pod \"50000cc4-9bd6-4cfb-a85f-15b0ae191071\" (UID: \"50000cc4-9bd6-4cfb-a85f-15b0ae191071\") " Nov 03 22:45:23.114609 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.114587 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50000cc4-9bd6-4cfb-a85f-15b0ae191071-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50000cc4-9bd6-4cfb-a85f-15b0ae191071-maas-api-service-ca\") pod \"50000cc4-9bd6-4cfb-a85f-15b0ae191071\" (UID: \"50000cc4-9bd6-4cfb-a85f-15b0ae191071\") " Nov 03 22:45:23.115405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.115382 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjg2z\" (UniqueName: \"kubernetes.io/projected/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17-kube-api-access-cjg2z\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.116088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.115973 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50000cc4-9bd6-4cfb-a85f-15b0ae191071-maas-api-service-ca" pod "50000cc4-9bd6-4cfb-a85f-15b0ae191071" (UID: "50000cc4-9bd6-4cfb-a85f-15b0ae191071"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:23.118408 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.118392 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50000cc4-9bd6-4cfb-a85f-15b0ae191071-kube-api-access-ml2fk" pod "50000cc4-9bd6-4cfb-a85f-15b0ae191071" (UID: "50000cc4-9bd6-4cfb-a85f-15b0ae191071"). InnerVolumeSpecName "kube-api-access-ml2fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:23.217867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.216947 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f863b3-9457-4888-b046-3cfd408e504a-kube-api-access-9nzz9\" (UniqueName: \"kubernetes.io/projected/81f863b3-9457-4888-b046-3cfd408e504a-kube-api-access-9nzz9\") pod \"81f863b3-9457-4888-b046-3cfd408e504a\" (UID: \"81f863b3-9457-4888-b046-3cfd408e504a\") " Nov 03 22:45:23.217867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.217097 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f863b3-9457-4888-b046-3cfd408e504a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f863b3-9457-4888-b046-3cfd408e504a-maas-api-service-ca\") pod \"81f863b3-9457-4888-b046-3cfd408e504a\" (UID: \"81f863b3-9457-4888-b046-3cfd408e504a\") " Nov 03 22:45:23.218266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.218242 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f863b3-9457-4888-b046-3cfd408e504a-maas-api-service-ca" pod "81f863b3-9457-4888-b046-3cfd408e504a" (UID: "81f863b3-9457-4888-b046-3cfd408e504a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:23.220869 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.220844 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml2fk\" (UniqueName: \"kubernetes.io/projected/50000cc4-9bd6-4cfb-a85f-15b0ae191071-kube-api-access-ml2fk\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.221056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.221035 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50000cc4-9bd6-4cfb-a85f-15b0ae191071-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.240728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.223735 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f863b3-9457-4888-b046-3cfd408e504a-kube-api-access-9nzz9" pod "81f863b3-9457-4888-b046-3cfd408e504a" (UID: "81f863b3-9457-4888-b046-3cfd408e504a"). InnerVolumeSpecName "kube-api-access-9nzz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:23.258704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.258381 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 03 22:45:23.258704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.258441 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 03 22:45:23.259330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.259223 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 03 22:45:23.259330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.259267 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 03 22:45:23.318800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.318763 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c029cafa-94ab-4b6a-b775-3fee6c1acea6" path="/var/lib/kubelet/pods/c029cafa-94ab-4b6a-b775-3fee6c1acea6/volumes" Nov 03 22:45:23.414153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.414070 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nzz9\" (UniqueName: \"kubernetes.io/projected/81f863b3-9457-4888-b046-3cfd408e504a-kube-api-access-9nzz9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.414542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.414511 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f863b3-9457-4888-b046-3cfd408e504a-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.415119 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.415078 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" podStartSLOduration=4.415065115 podStartE2EDuration="4.415065115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:23.412360872 +0000 UTC m=+2280.393094526" watchObservedRunningTime="2026-11-03 22:45:23.415065115 +0000 UTC m=+2280.395798767" Nov 03 22:45:23.615438 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.615396 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" Nov 03 22:45:23.617879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.617854 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-maas-api-service-ca\") pod \"ecc2af8c-c3b6-4bde-ac6f-398533e20acf\" (UID: \"ecc2af8c-c3b6-4bde-ac6f-398533e20acf\") " Nov 03 22:45:23.618031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.618004 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-kube-api-access-jtlms\" (UniqueName: \"kubernetes.io/projected/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-kube-api-access-jtlms\") pod \"ecc2af8c-c3b6-4bde-ac6f-398533e20acf\" (UID: \"ecc2af8c-c3b6-4bde-ac6f-398533e20acf\") " Nov 03 22:45:23.618676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.618630 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-maas-api-service-ca" pod "ecc2af8c-c3b6-4bde-ac6f-398533e20acf" (UID: "ecc2af8c-c3b6-4bde-ac6f-398533e20acf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:23.618895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.618880 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.619586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.619532 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" podStartSLOduration=4.619512005 podStartE2EDuration="4.619512005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:23.613532954 +0000 UTC m=+2280.594266608" watchObservedRunningTime="2026-11-03 22:45:23.619512005 +0000 UTC m=+2280.600245659" Nov 03 22:45:23.621492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.621468 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-kube-api-access-jtlms" pod "ecc2af8c-c3b6-4bde-ac6f-398533e20acf" (UID: "ecc2af8c-c3b6-4bde-ac6f-398533e20acf"). InnerVolumeSpecName "kube-api-access-jtlms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:23.685752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.684303 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" Nov 03 22:45:23.719508 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.719481 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3056f50-50c4-4804-af55-0e9e4f06f583-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3056f50-50c4-4804-af55-0e9e4f06f583-maas-api-service-ca\") pod \"f3056f50-50c4-4804-af55-0e9e4f06f583\" (UID: \"f3056f50-50c4-4804-af55-0e9e4f06f583\") " Nov 03 22:45:23.719811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.719789 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3056f50-50c4-4804-af55-0e9e4f06f583-maas-api-service-ca" pod "f3056f50-50c4-4804-af55-0e9e4f06f583" (UID: "f3056f50-50c4-4804-af55-0e9e4f06f583"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:23.719998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.719982 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3056f50-50c4-4804-af55-0e9e4f06f583-kube-api-access-cdkcz\" (UniqueName: \"kubernetes.io/projected/f3056f50-50c4-4804-af55-0e9e4f06f583-kube-api-access-cdkcz\") pod \"f3056f50-50c4-4804-af55-0e9e4f06f583\" (UID: \"f3056f50-50c4-4804-af55-0e9e4f06f583\") " Nov 03 22:45:23.720473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.720457 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3056f50-50c4-4804-af55-0e9e4f06f583-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.720567 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.720557 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtlms\" (UniqueName: \"kubernetes.io/projected/ecc2af8c-c3b6-4bde-ac6f-398533e20acf-kube-api-access-jtlms\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.722183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.722162 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3056f50-50c4-4804-af55-0e9e4f06f583-kube-api-access-cdkcz" pod "f3056f50-50c4-4804-af55-0e9e4f06f583" (UID: "f3056f50-50c4-4804-af55-0e9e4f06f583"). InnerVolumeSpecName "kube-api-access-cdkcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:23.811729 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:23.811683 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:23.822103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.822076 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdkcz\" (UniqueName: \"kubernetes.io/projected/f3056f50-50c4-4804-af55-0e9e4f06f583-kube-api-access-cdkcz\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.860711 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.860552 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:23.973467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.973434 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:45:23.973854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:23.973800 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:45:24.078667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.077878 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" event={"ID":"ecc2af8c-c3b6-4bde-ac6f-398533e20acf","Type":"ContainerDied","Data":"599451bdfedd4f713b55ff875811fa056222455f853ef706ab476fdf46fe16f4"} Nov 03 22:45:24.078667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.077948 4298 scope.go:122] "RemoveContainer" containerID="9d6489c1fda8636a683c1466b558c4350edd65a972c4cdc2fc772cebea832084" Nov 03 22:45:24.078667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.078112 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6" Nov 03 22:45:24.094933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.094901 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:24.167913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.165244 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" event={"ID":"f3056f50-50c4-4804-af55-0e9e4f06f583","Type":"ContainerDied","Data":"d7ca4bf48b3b757e975d4712a0b454844d982152995c516866a32136c1acba03"} Nov 03 22:45:24.167913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.165376 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp" Nov 03 22:45:24.209066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.209033 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr"] Nov 03 22:45:24.236971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.236884 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" event={"ID":"81f863b3-9457-4888-b046-3cfd408e504a","Type":"ContainerDied","Data":"d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef"} Nov 03 22:45:24.237300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.237279 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr" Nov 03 22:45:24.260169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.260140 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 03 22:45:24.260427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.260397 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 03 22:45:24.260965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.260943 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 03 22:45:24.261127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.261106 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 03 22:45:24.263510 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.263489 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6" Nov 03 22:45:24.288331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.265061 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn" Nov 03 22:45:24.348395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.348206 4298 scope.go:122] "RemoveContainer" containerID="c6253c25b5215a0a5b36c5f4519986f99cafd9e2dbc87f7d8c1c8f46b2299840" Nov 03 22:45:24.398665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.370338 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:24.398665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.370529 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b2fb1-29895765-m2nnr"] Nov 03 22:45:24.398665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.378498 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" podStartSLOduration=5.378479049 podStartE2EDuration="5.378479049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:24.233116103 +0000 UTC m=+2281.213849758" watchObservedRunningTime="2026-11-03 22:45:24.378479049 +0000 UTC m=+2281.359212703" Nov 03 22:45:24.562750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.562714 4298 scope.go:122] "RemoveContainer" containerID="31a7e71753b3794a9f81ff3c225c9de3dac22b64d5af873786186f648846e160" Nov 03 22:45:24.584004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.583958 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:24.659035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.659002 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght"] Nov 03 22:45:24.863098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.860750 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:24.918745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.918708 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-vbght"] Nov 03 22:45:24.968958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.968905 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:45:24.969259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:24.969207 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:45:25.071760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.071693 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd"] Nov 03 22:45:25.205273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.205243 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" path="/var/lib/kubelet/pods/03b37944-6808-42f4-857f-57ea8635a2cd/volumes" Nov 03 22:45:25.206202 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.206178 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" path="/var/lib/kubelet/pods/157ee087-43db-4f2f-879d-a6e477ddf041/volumes" Nov 03 22:45:25.206848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.206830 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f61714-29895765-6dqfd"] Nov 03 22:45:25.248492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.248041 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 03 22:45:25.248492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.248100 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 03 22:45:25.259249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.259138 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 03 22:45:25.259249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.259191 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 03 22:45:25.480724 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.479685 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn"] Nov 03 22:45:25.720737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.720708 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-345da9-29895765-pdsnn"] Nov 03 22:45:25.815766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.815637 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp"] Nov 03 22:45:25.843674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.826709 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c8f168-29895765-bpgjp"] Nov 03 22:45:25.920565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.920535 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6"] Nov 03 22:45:25.989673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.986086 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:45:26.023059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.022965 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:45:26.023290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:25.994670 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2a6801-29895765-md4p6"] Nov 03 22:45:26.112739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.110437 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr"] Nov 03 22:45:26.118719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.118251 4298 scope.go:122] "RemoveContainer" containerID="0644a1f61a8f5ce40e09f20354addac250027fbb01359604f38f1230e04cda24" Nov 03 22:45:26.120661 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.119856 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-97965e-29895765-fd6mr"] Nov 03 22:45:26.209683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.207725 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6"] Nov 03 22:45:26.213665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.213617 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f881e3-29895765-bkdw6"] Nov 03 22:45:26.217010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.216829 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:26.294686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.267383 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 03 22:45:26.294686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.267439 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 03 22:45:26.294686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.267868 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 03 22:45:26.294686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.267902 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 03 22:45:26.400976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.399080 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:26.604499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.604472 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:26.964056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.964021 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:45:26.964678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:26.964616 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:45:27.194838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.157536 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:27.221087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.216214 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1534c655-737d-499d-8fcf-00e85c96854c" path="/var/lib/kubelet/pods/1534c655-737d-499d-8fcf-00e85c96854c/volumes" Nov 03 22:45:27.221087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.216924 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" path="/var/lib/kubelet/pods/50000cc4-9bd6-4cfb-a85f-15b0ae191071/volumes" Nov 03 22:45:27.221087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.217472 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f863b3-9457-4888-b046-3cfd408e504a" path="/var/lib/kubelet/pods/81f863b3-9457-4888-b046-3cfd408e504a/volumes" Nov 03 22:45:27.221087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.219188 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" path="/var/lib/kubelet/pods/9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17/volumes" Nov 03 22:45:27.223441 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.223417 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" path="/var/lib/kubelet/pods/ecc2af8c-c3b6-4bde-ac6f-398533e20acf/volumes" Nov 03 22:45:27.237736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.224281 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" path="/var/lib/kubelet/pods/f3056f50-50c4-4804-af55-0e9e4f06f583/volumes" Nov 03 22:45:27.248668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.240482 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 03 22:45:27.248668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.240540 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 03 22:45:27.263685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.261754 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 03 22:45:27.263685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.261814 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 03 22:45:27.406047 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.402587 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:27.595719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.595571 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:27.950743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:27.948630 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:28.014539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.014093 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:45:28.014539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.014151 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:45:28.240322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.240228 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 03 22:45:28.240322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.240296 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 03 22:45:28.253576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.253541 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 03 22:45:28.253724 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.253600 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 03 22:45:28.270352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.270321 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:28.767914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.767873 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c"] Nov 03 22:45:28.768956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.768929 4298 cpu_manager.go:414] "containerMap: removing container" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerName="cleanup" Nov 03 22:45:28.768956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.768957 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.768980 4298 cpu_manager.go:414] "containerMap: removing container" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.768989 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769007 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1534c655-737d-499d-8fcf-00e85c96854c" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769016 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1534c655-737d-499d-8fcf-00e85c96854c" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769027 4298 cpu_manager.go:414] "containerMap: removing container" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769036 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769055 4298 cpu_manager.go:414] "containerMap: removing container" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769062 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769071 4298 cpu_manager.go:414] "containerMap: removing container" podUID="81f863b3-9457-4888-b046-3cfd408e504a" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769078 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f863b3-9457-4888-b046-3cfd408e504a" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769091 4298 cpu_manager.go:414] "containerMap: removing container" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769099 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769118 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769125 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769143 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerName="cleanup" Nov 03 22:45:28.769148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769151 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769160 4298 cpu_manager.go:414] "containerMap: removing container" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769168 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769183 4298 cpu_manager.go:414] "containerMap: removing container" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769191 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769204 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769211 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769226 4298 cpu_manager.go:414] "containerMap: removing container" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769234 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769252 4298 cpu_manager.go:414] "containerMap: removing container" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769259 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769271 4298 cpu_manager.go:414] "containerMap: removing container" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769278 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769303 4298 cpu_manager.go:414] "containerMap: removing container" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769311 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769321 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769327 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769346 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769353 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769364 4298 cpu_manager.go:414] "containerMap: removing container" podUID="81f863b3-9457-4888-b046-3cfd408e504a" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769372 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f863b3-9457-4888-b046-3cfd408e504a" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769383 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1534c655-737d-499d-8fcf-00e85c96854c" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769389 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1534c655-737d-499d-8fcf-00e85c96854c" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769531 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769545 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769556 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769565 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f863b3-9457-4888-b046-3cfd408e504a" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769574 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1534c655-737d-499d-8fcf-00e85c96854c" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769584 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769594 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769604 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1534c655-737d-499d-8fcf-00e85c96854c" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769612 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769621 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769631 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f863b3-9457-4888-b046-3cfd408e504a" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769719 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769736 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b37944-6808-42f4-857f-57ea8635a2cd" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769750 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769758 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769768 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerName="cleanup" Nov 03 22:45:28.770007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.769779 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" containerName="cleanup" Nov 03 22:45:28.776565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.776525 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:28.890668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.889918 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-295682-openshift-default-dockercfg-f6nsj\"" Nov 03 22:45:28.915667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.895423 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c"] Nov 03 22:45:28.978666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.978099 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-podinfo\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:28.978666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.978174 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-envoy\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:28.978666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.978281 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-token\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:28.978666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.978342 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttb9n\" (UniqueName: \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-kube-api-access-ttb9n\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:28.978666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.978413 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36642f0e-de5f-4793-9398-0965f2d8ea70-istiod-ca-cert\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:28.978666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.978516 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-data\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:28.978666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.978582 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-socket\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:28.978666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.978604 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-certs\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:28.978666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.978636 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-credential-socket\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.009694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.980344 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:45:29.009694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:28.980406 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:45:29.105234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.105081 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36642f0e-de5f-4793-9398-0965f2d8ea70-istiod-ca-cert\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.105934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.105904 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-data\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.106177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.106139 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-socket\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.106513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.106296 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-certs\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.106700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.106680 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-credential-socket\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.106904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.106886 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-podinfo\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.107074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.107046 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-envoy\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.107318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.107302 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-token\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.107731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.107481 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttb9n\" (UniqueName: \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-kube-api-access-ttb9n\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.111691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.111108 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-credential-socket\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.113853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.113802 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:29.117167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.116939 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36642f0e-de5f-4793-9398-0965f2d8ea70-istiod-ca-cert\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.118240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.118218 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-data\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.120968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.120948 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-socket\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.121375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.121356 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-certs\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.187678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.187571 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttb9n\" (UniqueName: \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-kube-api-access-ttb9n\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.191720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.191553 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-token\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.192943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.192843 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-envoy\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.195860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.195772 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-podinfo\") pod \"e2e-api-b-295682-openshift-default-7446766979-95n6c\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.241210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.241167 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 03 22:45:29.241419 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.241247 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 03 22:45:29.253681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.253615 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 03 22:45:29.253826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.253706 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 03 22:45:29.394397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.394181 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:29.477286 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.477248 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:29.974184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.974143 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c"] Nov 03 22:45:29.975128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.975079 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:29.975256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.975161 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:29.975256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:29.975235 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:30.186806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:30.185460 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" event={"ID":"36642f0e-de5f-4793-9398-0965f2d8ea70","Type":"ContainerStarted","Data":"25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294"} Nov 03 22:45:30.186806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:30.185524 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" event={"ID":"36642f0e-de5f-4793-9398-0965f2d8ea70","Type":"ContainerStarted","Data":"9eee5dc2b6b981efe5978255273df5948e27d3bda0f46a46342628c5242994b4"} Nov 03 22:45:30.240725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:30.240455 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 03 22:45:30.240725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:30.240545 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 03 22:45:30.316667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:30.312840 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" podStartSLOduration=2.312819277 podStartE2EDuration="2.312819277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:30.311239867 +0000 UTC m=+2287.291973524" watchObservedRunningTime="2026-11-03 22:45:30.312819277 +0000 UTC m=+2287.293552933" Nov 03 22:45:30.478666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:30.477732 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:31.040680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.021746 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:31.040680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.021809 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:31.234667 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:31.233845 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:31.298720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.297760 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:31.298720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.297829 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:31.339706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.339670 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:31.340388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.340348 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:31.342041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.341989 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:45:31.487207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.487169 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:31.553072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.552988 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:31.553582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.553452 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:31.554472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.554408 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:45:31.677877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.677844 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:31.678183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.678158 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:31.679803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:31.679752 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:45:32.204832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:32.204801 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:32.205845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:32.205823 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:45:32.371682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:32.370857 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:32.568868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:32.568789 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:32.832550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:32.832472 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:33.023931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:33.023850 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:33.203896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:33.203865 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:33.525722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:33.512530 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:37.043906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:37.043873 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:37.235942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:37.235879 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:38.900727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:38.900638 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:39.221421 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:39.221355 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:41.253029 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:41.252987 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:44.382754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:44.382719 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:47.575177 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:47.575141 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:45:48.820886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:48.820837 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:49.540031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:49.539997 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:50.974244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:50.974182 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:51.152759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:51.152725 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:51.349133 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:51.349040 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:53.448628 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:53.448591 4298 generic.go:356] "Generic (PLEG): container finished" podID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerID="5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726" exitCode=0 Nov 03 22:45:53.449281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:53.448668 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" event={"ID":"e6c3301f-7e55-4e81-b5ce-420873ef1242","Type":"ContainerDied","Data":"5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726"} Nov 03 22:45:53.449281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:53.449108 4298 scope.go:122] "RemoveContainer" containerID="5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726" Nov 03 22:45:53.450832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:53.450806 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:53.450895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:53.450874 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:53.450931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:53.450909 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:53.811613 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:45:53.811526 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:54.460803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:54.460771 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" event={"ID":"e6c3301f-7e55-4e81-b5ce-420873ef1242","Type":"ContainerStarted","Data":"37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979"} Nov 03 22:45:55.123174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:55.123143 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:55.245423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:55.245388 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:45:55.245849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:55.245825 4298 patch_prober.go:29] interesting pod/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 03 22:45:55.246042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:55.246016 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 03 22:45:55.346523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:55.346485 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:55.544594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:55.544539 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:55.748992 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:55.748950 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:55.949937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:55.949898 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:56.248879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:56.248773 4298 patch_prober.go:29] interesting pod/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 03 22:45:56.248879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:56.248852 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 03 22:45:57.230058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.230018 4298 patch_prober.go:29] interesting pod/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 03 22:45:57.230706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.230666 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 03 22:45:57.456879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.456839 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48"] Nov 03 22:45:57.457535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.457516 4298 cpu_manager.go:414] "containerMap: removing container" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerName="cleanup" Nov 03 22:45:57.457726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.457712 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerName="cleanup" Nov 03 22:45:57.457813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.457806 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1534c655-737d-499d-8fcf-00e85c96854c" containerName="cleanup" Nov 03 22:45:57.457862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.457856 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1534c655-737d-499d-8fcf-00e85c96854c" containerName="cleanup" Nov 03 22:45:57.457919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.457913 4298 cpu_manager.go:414] "containerMap: removing container" podUID="81f863b3-9457-4888-b046-3cfd408e504a" containerName="cleanup" Nov 03 22:45:57.457977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.457967 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f863b3-9457-4888-b046-3cfd408e504a" containerName="cleanup" Nov 03 22:45:57.458100 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.458091 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerName="cleanup" Nov 03 22:45:57.458155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.458148 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerName="cleanup" Nov 03 22:45:57.458349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.458341 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1534c655-737d-499d-8fcf-00e85c96854c" containerName="cleanup" Nov 03 22:45:57.458409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.458403 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bdf1a2b-7d0b-4a5d-8b7b-433d6b9ceb17" containerName="cleanup" Nov 03 22:45:57.458465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.458458 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f863b3-9457-4888-b046-3cfd408e504a" containerName="cleanup" Nov 03 22:45:57.458532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.458523 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc2af8c-c3b6-4bde-ac6f-398533e20acf" containerName="cleanup" Nov 03 22:45:57.458673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.458663 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="157ee087-43db-4f2f-879d-a6e477ddf041" containerName="cleanup" Nov 03 22:45:57.458766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.458757 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3056f50-50c4-4804-af55-0e9e4f06f583" containerName="cleanup" Nov 03 22:45:57.458834 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.458826 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="50000cc4-9bd6-4cfb-a85f-15b0ae191071" containerName="cleanup" Nov 03 22:45:57.462755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.462735 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.518111 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.518032 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9295625-3631-4888-b199-02edcc9a4308-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.518289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.518153 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.518289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.518205 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.518289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.518268 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9295625-3631-4888-b199-02edcc9a4308-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.518414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.518303 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.518414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.518344 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.518414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.518401 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.518518 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.518416 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.518518 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.518466 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kffh6\" (UniqueName: \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-kube-api-access-kffh6\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.563581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.563531 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48"] Nov 03 22:45:57.619601 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.619565 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9295625-3631-4888-b199-02edcc9a4308-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.619601 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.619602 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.619873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.619635 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.619873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.619686 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9295625-3631-4888-b199-02edcc9a4308-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.619873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.619814 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.620064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.619894 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.620064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.619999 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.620064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.620005 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.620064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.620046 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.620282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.620129 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kffh6\" (UniqueName: \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-kube-api-access-kffh6\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.620282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.620187 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.620282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.620185 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.620588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.620560 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9295625-3631-4888-b199-02edcc9a4308-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.620632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.620584 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.622237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.622200 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.622798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.622781 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9295625-3631-4888-b199-02edcc9a4308-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.676793 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.676760 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.677196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.677179 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kffh6\" (UniqueName: \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-kube-api-access-kffh6\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:57.777743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:57.777671 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:58.261905 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:58.253117 4298 patch_prober.go:29] interesting pod/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 03 22:45:58.261905 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:58.253180 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 03 22:45:58.342084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:58.334478 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48"] Nov 03 22:45:58.368189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:58.368142 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:58.368496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:58.368469 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:58.368705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:58.368684 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:45:58.495807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:58.495769 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" event={"ID":"c9295625-3631-4888-b199-02edcc9a4308","Type":"ContainerStarted","Data":"011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501"} Nov 03 22:45:58.495958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:58.495837 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" event={"ID":"c9295625-3631-4888-b199-02edcc9a4308","Type":"ContainerStarted","Data":"e840e930921362e0c83df5e4dbd2fae4754f368bc9d3cd933a147fc862c02e02"} Nov 03 22:45:58.623690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:58.623578 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" podStartSLOduration=1.6235654149999998 podStartE2EDuration="1.623565415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:58.621305231 +0000 UTC m=+2315.602038881" watchObservedRunningTime="2026-11-03 22:45:58.623565415 +0000 UTC m=+2315.604299121" Nov 03 22:45:58.777989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:58.777958 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:45:58.895593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:58.895520 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:45:59.239667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:59.236105 4298 patch_prober.go:29] interesting pod/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 03 22:45:59.239667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:59.236174 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 03 22:45:59.779582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:59.779547 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 22:45:59.780009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:45:59.779602 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" podUID="c9295625-3631-4888-b199-02edcc9a4308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 22:46:00.080393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.080308 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7"] Nov 03 22:46:00.084759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.084740 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:00.109602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.109574 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2225ce43-7jfr7\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:00.109739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.109616 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2225ce43-7jfr7\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:00.109739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.109705 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvm8\" (UniqueName: \"kubernetes.io/projected/6983c091-4d4a-4001-84cb-b89bcef6a2f3-kube-api-access-wcvm8\") pod \"maas-api-revoke-sub-2225ce43-7jfr7\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:00.157259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.157218 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7"] Nov 03 22:46:00.186180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.186147 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:46:00.186180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.186187 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:46:00.194809 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.194774 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24"] Nov 03 22:46:00.200338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.200311 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:00.211179 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.211146 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2225ce43-7jfr7\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:00.211561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.211540 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2225ce43-7jfr7\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:00.212414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.211852 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvm8\" (UniqueName: \"kubernetes.io/projected/6983c091-4d4a-4001-84cb-b89bcef6a2f3-kube-api-access-wcvm8\") pod \"maas-api-revoke-sub-2225ce43-7jfr7\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:00.212414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.211897 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2225ce43-7jfr7\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:00.214099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.214078 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2225ce43-7jfr7\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:00.255454 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.255421 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24"] Nov 03 22:46:00.258515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.258489 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:46:00.259579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.259557 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:46:00.269121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.269094 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvm8\" (UniqueName: \"kubernetes.io/projected/6983c091-4d4a-4001-84cb-b89bcef6a2f3-kube-api-access-wcvm8\") pod \"maas-api-revoke-sub-2225ce43-7jfr7\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:00.313364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.313324 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8br4g\" (UniqueName: \"kubernetes.io/projected/3538021b-3161-486c-b39d-ebccba8905af-kube-api-access-8br4g\") pod \"maas-api-revoke-sub-087bac71-n5s24\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:00.313540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.313482 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3538021b-3161-486c-b39d-ebccba8905af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-087bac71-n5s24\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:00.313603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.313572 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3538021b-3161-486c-b39d-ebccba8905af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-087bac71-n5s24\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:00.354219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.354144 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9"] Nov 03 22:46:00.358734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.358715 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:00.399173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.399142 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:00.405138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.405110 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x"] Nov 03 22:46:00.410730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.410710 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:00.412803 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.412779 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9"] Nov 03 22:46:00.421988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.421960 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8br4g\" (UniqueName: \"kubernetes.io/projected/3538021b-3161-486c-b39d-ebccba8905af-kube-api-access-8br4g\") pod \"maas-api-revoke-sub-087bac71-n5s24\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:00.423372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.422119 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqt9s\" (UniqueName: \"kubernetes.io/projected/d2f9ff8b-8274-4fcc-abd7-a28523f93055-kube-api-access-dqt9s\") pod \"maas-api-revoke-sub-0388afdf-tjdw9\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:00.423372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.422160 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3538021b-3161-486c-b39d-ebccba8905af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-087bac71-n5s24\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:00.423372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.422243 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3538021b-3161-486c-b39d-ebccba8905af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-087bac71-n5s24\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:00.423372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.422531 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0388afdf-tjdw9\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:00.423372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.422598 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0388afdf-tjdw9\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:00.423765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.423534 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3538021b-3161-486c-b39d-ebccba8905af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-087bac71-n5s24\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:00.450230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.450200 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3538021b-3161-486c-b39d-ebccba8905af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-087bac71-n5s24\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:00.491759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.491727 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8br4g\" (UniqueName: \"kubernetes.io/projected/3538021b-3161-486c-b39d-ebccba8905af-kube-api-access-8br4g\") pod \"maas-api-revoke-sub-087bac71-n5s24\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:00.494244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.494218 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x"] Nov 03 22:46:00.514440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.514408 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:00.541391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.541363 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqt9s\" (UniqueName: \"kubernetes.io/projected/d2f9ff8b-8274-4fcc-abd7-a28523f93055-kube-api-access-dqt9s\") pod \"maas-api-revoke-sub-0388afdf-tjdw9\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:00.541789 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.541765 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0388afdf-tjdw9\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:00.541975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.541956 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjqv9\" (UniqueName: \"kubernetes.io/projected/e9c8f51f-60ce-44a7-8e67-310faeae985b-kube-api-access-jjqv9\") pod \"maas-api-revoke-sub-29b25d69-7cl9x\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:00.542146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.542123 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0388afdf-tjdw9\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:00.542319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.542293 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b25d69-7cl9x\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:00.542459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.542442 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b25d69-7cl9x\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:00.544867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.544839 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0388afdf-tjdw9\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:00.566394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.566346 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0388afdf-tjdw9\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:00.619616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.617921 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqt9s\" (UniqueName: \"kubernetes.io/projected/d2f9ff8b-8274-4fcc-abd7-a28523f93055-kube-api-access-dqt9s\") pod \"maas-api-revoke-sub-0388afdf-tjdw9\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:00.652997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.652329 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjqv9\" (UniqueName: \"kubernetes.io/projected/e9c8f51f-60ce-44a7-8e67-310faeae985b-kube-api-access-jjqv9\") pod \"maas-api-revoke-sub-29b25d69-7cl9x\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:00.652997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.652441 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b25d69-7cl9x\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:00.652997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.652491 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b25d69-7cl9x\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:00.658480 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.655485 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b25d69-7cl9x\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:00.664364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.664336 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b25d69-7cl9x\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:00.674046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.674024 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:00.713412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.713344 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjqv9\" (UniqueName: \"kubernetes.io/projected/e9c8f51f-60ce-44a7-8e67-310faeae985b-kube-api-access-jjqv9\") pod \"maas-api-revoke-sub-29b25d69-7cl9x\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:00.778797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.778762 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 22:46:00.778974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.778828 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" podUID="c9295625-3631-4888-b199-02edcc9a4308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 22:46:00.790945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.790923 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:00.874407 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.874360 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:00.874728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:00.874706 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7"] Nov 03 22:46:00.876592 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:46:00.876556 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6983c091_4d4a_4001_84cb_b89bcef6a2f3.slice/crio-3b36c06bf164787d9bdf35592ac20736b51b86b338e7ce6d88701acc9eb2c9b0 WatchSource:0}: Error finding container 3b36c06bf164787d9bdf35592ac20736b51b86b338e7ce6d88701acc9eb2c9b0: Status 404 returned error can't find the container with id 3b36c06bf164787d9bdf35592ac20736b51b86b338e7ce6d88701acc9eb2c9b0 Nov 03 22:46:01.077618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.077505 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:01.093170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.093138 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24"] Nov 03 22:46:01.289486 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:46:01.289133 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f9ff8b_8274_4fcc_abd7_a28523f93055.slice/crio-836a7c3ad7f7f7b737265ff1e2b3f4b302c6caaf3debe84cd8b194ee3f40c0d9 WatchSource:0}: Error finding container 836a7c3ad7f7f7b737265ff1e2b3f4b302c6caaf3debe84cd8b194ee3f40c0d9: Status 404 returned error can't find the container with id 836a7c3ad7f7f7b737265ff1e2b3f4b302c6caaf3debe84cd8b194ee3f40c0d9 Nov 03 22:46:01.304604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.304564 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9"] Nov 03 22:46:01.500915 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:46:01.488799 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache]" Nov 03 22:46:01.505718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.505682 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm"] Nov 03 22:46:01.534184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.534097 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:46:01.647030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.608050 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:01.647030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.612880 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-472b46-dockercfg-x9bgz\"" Nov 03 22:46:01.647030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.613060 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-472b46\"" Nov 03 22:46:01.647030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.639716 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6zcs\" (UniqueName: \"kubernetes.io/projected/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-kube-api-access-n6zcs\") pod \"payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm\" (UID: \"6ed647c8-3abf-4243-a464-cf1fbbef7d4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:46:01.647030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.639951 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm\" (UID: \"6ed647c8-3abf-4243-a464-cf1fbbef7d4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:46:01.699726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.699690 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x"] Nov 03 22:46:01.720368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.720230 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" event={"ID":"d2f9ff8b-8274-4fcc-abd7-a28523f93055","Type":"ContainerStarted","Data":"836a7c3ad7f7f7b737265ff1e2b3f4b302c6caaf3debe84cd8b194ee3f40c0d9"} Nov 03 22:46:01.760779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.756719 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6zcs\" (UniqueName: \"kubernetes.io/projected/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-kube-api-access-n6zcs\") pod \"payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm\" (UID: \"6ed647c8-3abf-4243-a464-cf1fbbef7d4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:46:01.760779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.756926 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm\" (UID: \"6ed647c8-3abf-4243-a464-cf1fbbef7d4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:46:01.760779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.758129 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm\" (UID: \"6ed647c8-3abf-4243-a464-cf1fbbef7d4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:46:01.764595 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.764488 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm"] Nov 03 22:46:01.765781 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.765711 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" event={"ID":"6983c091-4d4a-4001-84cb-b89bcef6a2f3","Type":"ContainerStarted","Data":"3b36c06bf164787d9bdf35592ac20736b51b86b338e7ce6d88701acc9eb2c9b0"} Nov 03 22:46:01.768596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.768565 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" event={"ID":"e9c8f51f-60ce-44a7-8e67-310faeae985b","Type":"ContainerStarted","Data":"027c5f5b463d88e967170d70913af2b1deed24839606a6d2095727c0ec2df1f6"} Nov 03 22:46:01.770247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.770185 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" event={"ID":"3538021b-3161-486c-b39d-ebccba8905af","Type":"ContainerStarted","Data":"42c94e00aba637ec9528b2c3ecd3629b96fee5f1d1b3784f8417602cfb4025f9"} Nov 03 22:46:01.771813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.771793 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj"] Nov 03 22:46:01.777256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.777234 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:46:01.778312 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.778291 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 22:46:01.778392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.778329 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" podUID="c9295625-3631-4888-b199-02edcc9a4308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 22:46:01.779683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.779667 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj"] Nov 03 22:46:01.853763 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.853728 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6zcs\" (UniqueName: \"kubernetes.io/projected/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-kube-api-access-n6zcs\") pod \"payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm\" (UID: \"6ed647c8-3abf-4243-a464-cf1fbbef7d4a\") " pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:46:01.859720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.859692 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj\" (UID: \"4f935ec2-2164-46e1-afe2-68fe6c5b4f24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:46:01.859973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.859954 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f274s\" (UniqueName: \"kubernetes.io/projected/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-kube-api-access-f274s\") pod \"payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj\" (UID: \"4f935ec2-2164-46e1-afe2-68fe6c5b4f24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:46:01.961661 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.961603 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj\" (UID: \"4f935ec2-2164-46e1-afe2-68fe6c5b4f24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:46:01.961820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.961702 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f274s\" (UniqueName: \"kubernetes.io/projected/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-kube-api-access-f274s\") pod \"payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj\" (UID: \"4f935ec2-2164-46e1-afe2-68fe6c5b4f24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:46:01.962399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:01.962379 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj\" (UID: \"4f935ec2-2164-46e1-afe2-68fe6c5b4f24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:46:02.019154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.019077 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f274s\" (UniqueName: \"kubernetes.io/projected/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-kube-api-access-f274s\") pod \"payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj\" (UID: \"4f935ec2-2164-46e1-afe2-68fe6c5b4f24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:46:02.073401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.073365 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:46:02.095042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.095011 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:46:02.476815 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.476782 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm"] Nov 03 22:46:02.510729 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:46:02.510683 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed647c8_3abf_4243_a464_cf1fbbef7d4a.slice/crio-ec6515f02dbd80ee92ac97172b02a9ef790ebbb585b8abd42effeef403ecddc9 WatchSource:0}: Error finding container ec6515f02dbd80ee92ac97172b02a9ef790ebbb585b8abd42effeef403ecddc9: Status 404 returned error can't find the container with id ec6515f02dbd80ee92ac97172b02a9ef790ebbb585b8abd42effeef403ecddc9 Nov 03 22:46:02.543768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.538957 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj"] Nov 03 22:46:02.557119 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:46:02.557072 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f935ec2_2164_46e1_afe2_68fe6c5b4f24.slice/crio-d36cb49e37cb1bc652c0bf58dadf40fc6944c4e77fe09f76bddf25d81a6753f1 WatchSource:0}: Error finding container d36cb49e37cb1bc652c0bf58dadf40fc6944c4e77fe09f76bddf25d81a6753f1: Status 404 returned error can't find the container with id d36cb49e37cb1bc652c0bf58dadf40fc6944c4e77fe09f76bddf25d81a6753f1 Nov 03 22:46:02.778256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.778223 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 22:46:02.778450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.778271 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" podUID="c9295625-3631-4888-b199-02edcc9a4308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 22:46:02.779429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.779407 4298 generic.go:356] "Generic (PLEG): container finished" podID="d2f9ff8b-8274-4fcc-abd7-a28523f93055" containerID="f41c80503316333fa3fe09b4ca70f0837d5a57ee0c2e23a0089bdd2f34149adc" exitCode=0 Nov 03 22:46:02.779527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.779503 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" event={"ID":"d2f9ff8b-8274-4fcc-abd7-a28523f93055","Type":"ContainerDied","Data":"f41c80503316333fa3fe09b4ca70f0837d5a57ee0c2e23a0089bdd2f34149adc"} Nov 03 22:46:02.781304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.781278 4298 generic.go:356] "Generic (PLEG): container finished" podID="6983c091-4d4a-4001-84cb-b89bcef6a2f3" containerID="d1de19adf5744252c62b757b7c32acd493536cf9e76da0bd7d519bf176b7dd41" exitCode=0 Nov 03 22:46:02.781445 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.781313 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" event={"ID":"6983c091-4d4a-4001-84cb-b89bcef6a2f3","Type":"ContainerDied","Data":"d1de19adf5744252c62b757b7c32acd493536cf9e76da0bd7d519bf176b7dd41"} Nov 03 22:46:02.783162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.783134 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" event={"ID":"e9c8f51f-60ce-44a7-8e67-310faeae985b","Type":"ContainerStarted","Data":"eb9941dba1e634189325fbb69e95dd7e11af82ae2fc23d9a3c0106cd8c151cbb"} Nov 03 22:46:02.784674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.784637 4298 generic.go:356] "Generic (PLEG): container finished" podID="3538021b-3161-486c-b39d-ebccba8905af" containerID="0862162122b750a78d55383c2e4a4b261e68dd026be50ee7d68814597bdc064c" exitCode=0 Nov 03 22:46:02.784771 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.784731 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" event={"ID":"3538021b-3161-486c-b39d-ebccba8905af","Type":"ContainerDied","Data":"0862162122b750a78d55383c2e4a4b261e68dd026be50ee7d68814597bdc064c"} Nov 03 22:46:02.786298 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.786276 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" event={"ID":"6ed647c8-3abf-4243-a464-cf1fbbef7d4a","Type":"ContainerStarted","Data":"75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252"} Nov 03 22:46:02.786387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.786324 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:46:02.786387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.786345 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" event={"ID":"6ed647c8-3abf-4243-a464-cf1fbbef7d4a","Type":"ContainerStarted","Data":"ec6515f02dbd80ee92ac97172b02a9ef790ebbb585b8abd42effeef403ecddc9"} Nov 03 22:46:02.787820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.787797 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" event={"ID":"4f935ec2-2164-46e1-afe2-68fe6c5b4f24","Type":"ContainerStarted","Data":"b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d"} Nov 03 22:46:02.787909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.787852 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:46:02.787909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:02.787871 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" event={"ID":"4f935ec2-2164-46e1-afe2-68fe6c5b4f24","Type":"ContainerStarted","Data":"d36cb49e37cb1bc652c0bf58dadf40fc6944c4e77fe09f76bddf25d81a6753f1"} Nov 03 22:46:03.049911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:03.045906 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" podStartSLOduration=2.045882406 podStartE2EDuration="2.045882406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:03.023372516 +0000 UTC m=+2320.004106170" watchObservedRunningTime="2026-11-03 22:46:03.045882406 +0000 UTC m=+2320.026616060" Nov 03 22:46:03.167885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:03.167773 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" podStartSLOduration=2.167758616 podStartE2EDuration="2.167758616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:03.163089234 +0000 UTC m=+2320.143822888" watchObservedRunningTime="2026-11-03 22:46:03.167758616 +0000 UTC m=+2320.148492269" Nov 03 22:46:03.512581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:03.512527 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" podStartSLOduration=3.5125108149999997 podStartE2EDuration="3.512510815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:03.509444829 +0000 UTC m=+2320.490178481" watchObservedRunningTime="2026-11-03 22:46:03.512510815 +0000 UTC m=+2320.493244497" Nov 03 22:46:03.778938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:03.778845 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 22:46:03.778938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:03.778906 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" podUID="c9295625-3631-4888-b199-02edcc9a4308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 22:46:03.802607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:03.802561 4298 generic.go:356] "Generic (PLEG): container finished" podID="e9c8f51f-60ce-44a7-8e67-310faeae985b" containerID="eb9941dba1e634189325fbb69e95dd7e11af82ae2fc23d9a3c0106cd8c151cbb" exitCode=0 Nov 03 22:46:03.803735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:03.803699 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" event={"ID":"e9c8f51f-60ce-44a7-8e67-310faeae985b","Type":"ContainerDied","Data":"eb9941dba1e634189325fbb69e95dd7e11af82ae2fc23d9a3c0106cd8c151cbb"} Nov 03 22:46:04.361109 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.361018 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:04.373708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.370782 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-openshift-ca\") pod \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " Nov 03 22:46:04.373708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.371015 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f9ff8b-8274-4fcc-abd7-a28523f93055-kube-api-access-dqt9s\" (UniqueName: \"kubernetes.io/projected/d2f9ff8b-8274-4fcc-abd7-a28523f93055-kube-api-access-dqt9s\") pod \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " Nov 03 22:46:04.373708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.371119 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-xks-ca\") pod \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\" (UID: \"d2f9ff8b-8274-4fcc-abd7-a28523f93055\") " Nov 03 22:46:04.373708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.371950 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-openshift-ca" pod "d2f9ff8b-8274-4fcc-abd7-a28523f93055" (UID: "d2f9ff8b-8274-4fcc-abd7-a28523f93055"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:04.378605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.376966 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-xks-ca" pod "d2f9ff8b-8274-4fcc-abd7-a28523f93055" (UID: "d2f9ff8b-8274-4fcc-abd7-a28523f93055"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:04.378605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.377142 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f9ff8b-8274-4fcc-abd7-a28523f93055-kube-api-access-dqt9s" pod "d2f9ff8b-8274-4fcc-abd7-a28523f93055" (UID: "d2f9ff8b-8274-4fcc-abd7-a28523f93055"). InnerVolumeSpecName "kube-api-access-dqt9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:04.379197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.379182 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:04.415931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.415902 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:04.473102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.473063 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-openshift-ca\") pod \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " Nov 03 22:46:04.473287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.473220 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-xks-ca\") pod \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " Nov 03 22:46:04.473287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.473262 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3538021b-3161-486c-b39d-ebccba8905af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3538021b-3161-486c-b39d-ebccba8905af-maas-api-openshift-ca\") pod \"3538021b-3161-486c-b39d-ebccba8905af\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " Nov 03 22:46:04.473392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.473297 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3538021b-3161-486c-b39d-ebccba8905af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3538021b-3161-486c-b39d-ebccba8905af-maas-api-xks-ca\") pod \"3538021b-3161-486c-b39d-ebccba8905af\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " Nov 03 22:46:04.473392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.473337 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3538021b-3161-486c-b39d-ebccba8905af-kube-api-access-8br4g\" (UniqueName: \"kubernetes.io/projected/3538021b-3161-486c-b39d-ebccba8905af-kube-api-access-8br4g\") pod \"3538021b-3161-486c-b39d-ebccba8905af\" (UID: \"3538021b-3161-486c-b39d-ebccba8905af\") " Nov 03 22:46:04.473493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.473412 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6983c091-4d4a-4001-84cb-b89bcef6a2f3-kube-api-access-wcvm8\" (UniqueName: \"kubernetes.io/projected/6983c091-4d4a-4001-84cb-b89bcef6a2f3-kube-api-access-wcvm8\") pod \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\" (UID: \"6983c091-4d4a-4001-84cb-b89bcef6a2f3\") " Nov 03 22:46:04.473580 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.473480 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-openshift-ca" pod "6983c091-4d4a-4001-84cb-b89bcef6a2f3" (UID: "6983c091-4d4a-4001-84cb-b89bcef6a2f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:04.473701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.473665 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3538021b-3161-486c-b39d-ebccba8905af-maas-api-openshift-ca" pod "3538021b-3161-486c-b39d-ebccba8905af" (UID: "3538021b-3161-486c-b39d-ebccba8905af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:04.474249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.474202 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:04.474347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.474252 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3538021b-3161-486c-b39d-ebccba8905af-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:04.474347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.474267 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9ff8b-8274-4fcc-abd7-a28523f93055-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:04.474347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.474281 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:04.474347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.474296 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqt9s\" (UniqueName: \"kubernetes.io/projected/d2f9ff8b-8274-4fcc-abd7-a28523f93055-kube-api-access-dqt9s\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:04.476045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.475776 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3538021b-3161-486c-b39d-ebccba8905af-kube-api-access-8br4g" pod "3538021b-3161-486c-b39d-ebccba8905af" (UID: "3538021b-3161-486c-b39d-ebccba8905af"). InnerVolumeSpecName "kube-api-access-8br4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:04.476045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.475840 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-xks-ca" pod "6983c091-4d4a-4001-84cb-b89bcef6a2f3" (UID: "6983c091-4d4a-4001-84cb-b89bcef6a2f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:04.476200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.476153 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3538021b-3161-486c-b39d-ebccba8905af-maas-api-xks-ca" pod "3538021b-3161-486c-b39d-ebccba8905af" (UID: "3538021b-3161-486c-b39d-ebccba8905af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:04.476200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.476153 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6983c091-4d4a-4001-84cb-b89bcef6a2f3-kube-api-access-wcvm8" pod "6983c091-4d4a-4001-84cb-b89bcef6a2f3" (UID: "6983c091-4d4a-4001-84cb-b89bcef6a2f3"). InnerVolumeSpecName "kube-api-access-wcvm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:04.576244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.576204 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6983c091-4d4a-4001-84cb-b89bcef6a2f3-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:04.576244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.576241 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3538021b-3161-486c-b39d-ebccba8905af-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:04.576244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.576251 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8br4g\" (UniqueName: \"kubernetes.io/projected/3538021b-3161-486c-b39d-ebccba8905af-kube-api-access-8br4g\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:04.576518 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.576261 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvm8\" (UniqueName: \"kubernetes.io/projected/6983c091-4d4a-4001-84cb-b89bcef6a2f3-kube-api-access-wcvm8\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:04.778953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.778919 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 22:46:04.779279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.779194 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" podUID="c9295625-3631-4888-b199-02edcc9a4308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 22:46:04.811440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.811406 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" Nov 03 22:46:04.811440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.811434 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9" event={"ID":"d2f9ff8b-8274-4fcc-abd7-a28523f93055","Type":"ContainerDied","Data":"836a7c3ad7f7f7b737265ff1e2b3f4b302c6caaf3debe84cd8b194ee3f40c0d9"} Nov 03 22:46:04.811677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.811484 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="836a7c3ad7f7f7b737265ff1e2b3f4b302c6caaf3debe84cd8b194ee3f40c0d9" Nov 03 22:46:04.813053 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.813034 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" Nov 03 22:46:04.813191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.813072 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7" event={"ID":"6983c091-4d4a-4001-84cb-b89bcef6a2f3","Type":"ContainerDied","Data":"3b36c06bf164787d9bdf35592ac20736b51b86b338e7ce6d88701acc9eb2c9b0"} Nov 03 22:46:04.813191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.813112 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b36c06bf164787d9bdf35592ac20736b51b86b338e7ce6d88701acc9eb2c9b0" Nov 03 22:46:04.814707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.814679 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" Nov 03 22:46:04.814707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.814694 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24" event={"ID":"3538021b-3161-486c-b39d-ebccba8905af","Type":"ContainerDied","Data":"42c94e00aba637ec9528b2c3ecd3629b96fee5f1d1b3784f8417602cfb4025f9"} Nov 03 22:46:04.814879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.814745 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42c94e00aba637ec9528b2c3ecd3629b96fee5f1d1b3784f8417602cfb4025f9" Nov 03 22:46:04.983712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:04.983680 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:05.082563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.082485 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-xks-ca\") pod \"e9c8f51f-60ce-44a7-8e67-310faeae985b\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " Nov 03 22:46:05.082743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.082661 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c8f51f-60ce-44a7-8e67-310faeae985b-kube-api-access-jjqv9\" (UniqueName: \"kubernetes.io/projected/e9c8f51f-60ce-44a7-8e67-310faeae985b-kube-api-access-jjqv9\") pod \"e9c8f51f-60ce-44a7-8e67-310faeae985b\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " Nov 03 22:46:05.082743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.082687 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-openshift-ca\") pod \"e9c8f51f-60ce-44a7-8e67-310faeae985b\" (UID: \"e9c8f51f-60ce-44a7-8e67-310faeae985b\") " Nov 03 22:46:05.083162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.083128 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-openshift-ca" pod "e9c8f51f-60ce-44a7-8e67-310faeae985b" (UID: "e9c8f51f-60ce-44a7-8e67-310faeae985b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:05.083462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.083433 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:05.084700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.084666 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-xks-ca" pod "e9c8f51f-60ce-44a7-8e67-310faeae985b" (UID: "e9c8f51f-60ce-44a7-8e67-310faeae985b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:05.084874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.084847 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c8f51f-60ce-44a7-8e67-310faeae985b-kube-api-access-jjqv9" pod "e9c8f51f-60ce-44a7-8e67-310faeae985b" (UID: "e9c8f51f-60ce-44a7-8e67-310faeae985b"). InnerVolumeSpecName "kube-api-access-jjqv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:05.185008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.184977 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8f51f-60ce-44a7-8e67-310faeae985b-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:05.185008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.185007 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjqv9\" (UniqueName: \"kubernetes.io/projected/e9c8f51f-60ce-44a7-8e67-310faeae985b-kube-api-access-jjqv9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:05.827868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.827816 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" Nov 03 22:46:05.827868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.827847 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x" event={"ID":"e9c8f51f-60ce-44a7-8e67-310faeae985b","Type":"ContainerDied","Data":"027c5f5b463d88e967170d70913af2b1deed24839606a6d2095727c0ec2df1f6"} Nov 03 22:46:05.828442 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:05.827908 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="027c5f5b463d88e967170d70913af2b1deed24839606a6d2095727c0ec2df1f6" Nov 03 22:46:06.049068 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:06.049036 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:46:06.049463 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:06.049445 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:46:06.050383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:06.050362 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:46:06.191208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:06.191131 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:06.378669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:06.378626 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:07.232672 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:07.232614 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:08.863163 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:46:08.863120 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache]" Nov 03 22:46:10.286500 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:10.286464 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:10.461550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:10.461517 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:11.542533 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:46:11.542481 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:46:12.445749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.445717 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8"] Nov 03 22:46:12.446426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446411 4298 cpu_manager.go:414] "containerMap: removing container" podUID="6983c091-4d4a-4001-84cb-b89bcef6a2f3" containerName="revoke-keys" Nov 03 22:46:12.446472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446428 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6983c091-4d4a-4001-84cb-b89bcef6a2f3" containerName="revoke-keys" Nov 03 22:46:12.446472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446444 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3538021b-3161-486c-b39d-ebccba8905af" containerName="revoke-keys" Nov 03 22:46:12.446472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446449 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3538021b-3161-486c-b39d-ebccba8905af" containerName="revoke-keys" Nov 03 22:46:12.446472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446463 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d2f9ff8b-8274-4fcc-abd7-a28523f93055" containerName="revoke-keys" Nov 03 22:46:12.446472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446468 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f9ff8b-8274-4fcc-abd7-a28523f93055" containerName="revoke-keys" Nov 03 22:46:12.446636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446495 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e9c8f51f-60ce-44a7-8e67-310faeae985b" containerName="revoke-keys" Nov 03 22:46:12.446636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446501 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c8f51f-60ce-44a7-8e67-310faeae985b" containerName="revoke-keys" Nov 03 22:46:12.446636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446621 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c8f51f-60ce-44a7-8e67-310faeae985b" containerName="revoke-keys" Nov 03 22:46:12.446636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446633 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="6983c091-4d4a-4001-84cb-b89bcef6a2f3" containerName="revoke-keys" Nov 03 22:46:12.446636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446654 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3538021b-3161-486c-b39d-ebccba8905af" containerName="revoke-keys" Nov 03 22:46:12.446828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.446664 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f9ff8b-8274-4fcc-abd7-a28523f93055" containerName="revoke-keys" Nov 03 22:46:12.450714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.450692 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.470910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.470878 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:12.556915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.556885 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8"] Nov 03 22:46:12.594735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.594699 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmwgj\" (UniqueName: \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-kube-api-access-zmwgj\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.594914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.594768 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.594914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.594849 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.594914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.594895 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.595023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.594992 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.595110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.595068 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.595110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.595094 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.595220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.595194 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.595321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.595308 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697071 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697116 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697146 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697173 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697241 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697343 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697383 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmwgj\" (UniqueName: \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-kube-api-access-zmwgj\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697518 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697532 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697596 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697637 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.697975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.697939 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.698034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.698008 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.698097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.698028 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.699548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.699523 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.700016 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.699994 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.721610 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:46:12.721565 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:46:12.722738 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:46:12.722694 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:46:12.754800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.754763 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.754956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.754861 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmwgj\" (UniqueName: \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-kube-api-access-zmwgj\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:12.765998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:12.765973 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:13.192343 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:46:13.192308 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a07627c_ac0d_4c25_97ec_f60441a3a3b7.slice/crio-b2f2b20d0fff6590cd96ee41ae152bf112e7685b7a5f22d40a83ac9e9b26709d WatchSource:0}: Error finding container b2f2b20d0fff6590cd96ee41ae152bf112e7685b7a5f22d40a83ac9e9b26709d: Status 404 returned error can't find the container with id b2f2b20d0fff6590cd96ee41ae152bf112e7685b7a5f22d40a83ac9e9b26709d Nov 03 22:46:13.195769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:13.195736 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:46:13.195860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:13.195807 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:46:13.195860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:13.195837 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:46:13.197711 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:13.197688 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8"] Nov 03 22:46:13.807797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:13.807766 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:46:13.808272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:13.808141 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:46:13.901271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:13.901232 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" event={"ID":"2a07627c-ac0d-4c25-97ec-f60441a3a3b7","Type":"ContainerStarted","Data":"8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf"} Nov 03 22:46:13.901545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:13.901522 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" event={"ID":"2a07627c-ac0d-4c25-97ec-f60441a3a3b7","Type":"ContainerStarted","Data":"b2f2b20d0fff6590cd96ee41ae152bf112e7685b7a5f22d40a83ac9e9b26709d"} Nov 03 22:46:14.066113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:14.066009 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" podStartSLOduration=2.065988252 podStartE2EDuration="2.065988252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:14.060072211 +0000 UTC m=+2331.040805865" watchObservedRunningTime="2026-11-03 22:46:14.065988252 +0000 UTC m=+2331.046721905" Nov 03 22:46:14.162236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:14.162201 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:14.766566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:14.766534 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:14.767954 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:14.767930 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:46:14.768044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:14.767981 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" podUID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:46:15.767005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:15.766973 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:46:15.767393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:15.767022 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" podUID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:46:16.551270 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:16.551118 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:16.769663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:16.769145 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:46:16.769663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:16.769230 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" podUID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:46:17.594081 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:46:17.593958 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:46:17.777666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:17.777086 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:46:17.777666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:17.777170 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" podUID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:46:18.191613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:18.191570 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:18.767174 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:18.767140 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:46:18.767509 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:18.767465 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" podUID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:46:19.767477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:19.767435 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:46:19.767919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:19.767500 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" podUID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:46:21.107779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:21.107730 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:21.108350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:21.108251 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:21.109234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:21.109210 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:46:21.162355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:21.162321 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:21.659452 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:46:21.659413 4298 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice/crio-c8a9777f43e98ad2b89ad4f986831a277cfdaa0d64f2b372e6b9bc9a480e390c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50000cc4_9bd6_4cfb_a85f_15b0ae191071.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice/crio-d258906901fdcd3342ac0ab44421b7d401c07ad03553b5a7999a891022710eef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f863b3_9457_4888_b046_3cfd408e504a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:46:22.523413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:22.523372 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:22.756561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:22.756531 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:23.546667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:23.545551 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:26.574129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:26.574075 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:26.769978 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:26.769938 4298 scope.go:122] "RemoveContainer" containerID="2c2333068efd42e99089ad5f06c6b78582738abced2e628720225affc778df95" Nov 03 22:46:28.100532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:28.100496 4298 generic.go:356] "Generic (PLEG): container finished" podID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerID="a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c" exitCode=0 Nov 03 22:46:28.101102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:28.100573 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" event={"ID":"74935ed0-dfaa-4277-af6d-76b89f6e8970","Type":"ContainerDied","Data":"a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c"} Nov 03 22:46:28.101222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:28.101204 4298 scope.go:122] "RemoveContainer" containerID="a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c" Nov 03 22:46:28.103017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:28.102991 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:46:28.103091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:28.103062 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:46:28.103130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:28.103096 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:46:28.462486 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:28.462446 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:29.109970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:29.109937 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" event={"ID":"74935ed0-dfaa-4277-af6d-76b89f6e8970","Type":"ContainerStarted","Data":"15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6"} Nov 03 22:46:29.901552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:29.901523 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:46:29.902483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:29.902455 4298 patch_prober.go:29] interesting pod/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 22:46:29.902612 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:29.902516 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 22:46:30.900947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:30.900916 4298 patch_prober.go:29] interesting pod/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 22:46:30.901405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:30.900981 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 22:46:31.530784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:31.530743 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:46:31.901455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:31.901367 4298 patch_prober.go:29] interesting pod/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 22:46:31.901455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:31.901430 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 22:46:32.900858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:32.900823 4298 patch_prober.go:29] interesting pod/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 22:46:32.901047 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:32.900880 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 22:46:33.444666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:33.444412 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:33.901460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:33.901422 4298 patch_prober.go:29] interesting pod/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 22:46:33.901709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:33.901496 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 22:46:34.901623 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:34.901588 4298 patch_prober.go:29] interesting pod/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 22:46:34.902046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:34.901673 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 22:46:35.905403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:35.905371 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:46:35.906550 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:35.906526 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:46:38.437938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:38.437909 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:43.247127 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:43.247091 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:43.423682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:43.423602 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:43.717407 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:43.717375 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:47.551780 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:46:47.551737 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:46:48.349998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:48.349968 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:51.606903 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:51.606868 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:51.877275 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:51.877184 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:53.327007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:53.326972 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:58.149267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:58.149226 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:58.325924 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:58.325874 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:58.503194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:46:58.503158 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:03.458070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:03.457985 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:08.455978 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:08.455945 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:13.437829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:13.437792 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:14.527571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:14.527537 4298 generic.go:356] "Generic (PLEG): container finished" podID="fe800041-c967-421b-a414-2a6aebb1d69c" containerID="d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d" exitCode=0 Nov 03 22:47:14.528143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:14.527583 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" event={"ID":"fe800041-c967-421b-a414-2a6aebb1d69c","Type":"ContainerDied","Data":"d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d"} Nov 03 22:47:14.528143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:14.528022 4298 scope.go:122] "RemoveContainer" containerID="d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d" Nov 03 22:47:14.529820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:14.529785 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:47:14.529942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:14.529874 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:47:14.529942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:14.529926 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:47:15.558332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:15.558267 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" event={"ID":"fe800041-c967-421b-a414-2a6aebb1d69c","Type":"ContainerStarted","Data":"8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a"} Nov 03 22:47:15.849168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:15.848236 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:47:15.849168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:15.848948 4298 patch_prober.go:29] interesting pod/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 22:47:15.849168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:15.849003 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 22:47:16.849678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:16.849232 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:16.849678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:16.849446 4298 patch_prober.go:29] interesting pod/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 22:47:16.849678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:16.849527 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 22:47:17.578206 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:47:17.578167 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:47:17.848720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:17.845980 4298 patch_prober.go:29] interesting pod/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 22:47:17.848720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:17.846042 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 22:47:18.375552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:18.375513 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:18.561384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:18.561346 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:18.844731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:18.844692 4298 patch_prober.go:29] interesting pod/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 22:47:18.845038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:18.845011 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 22:47:19.850078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:19.850038 4298 patch_prober.go:29] interesting pod/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 22:47:19.850671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:19.850604 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 22:47:20.406678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:20.406616 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:47:20.871484 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:20.871445 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:47:20.872755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:20.872736 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:47:21.491275 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:21.491242 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:23.177900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:23.177863 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:23.412070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:23.412016 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:23.588260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:23.588230 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:26.802315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:26.802284 4298 scope.go:122] "RemoveContainer" containerID="79ccbf97a1a8aca1bef81b6db1e3d8e61e0ae75db8c87e6b076a031d438ff855" Nov 03 22:47:26.812801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:26.812778 4298 scope.go:122] "RemoveContainer" containerID="7583e1898a340c186f7b4d436d49aec3832786566bdf7c83a58471eab3cfd10e" Nov 03 22:47:26.822835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:26.822814 4298 scope.go:122] "RemoveContainer" containerID="df0d25fc68a35cac59b0afb1ff4ce99609a9a0b038c66dbfab14061d2a0842ed" Nov 03 22:47:26.833889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:26.833869 4298 scope.go:122] "RemoveContainer" containerID="71a8f9359c28dcbacf779f8fb7f8676c2d83c7a9845e47e9d368436d38c3409a" Nov 03 22:47:26.927564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:26.927530 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:28.313892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:28.313854 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:32.629776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:32.629743 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:32.735925 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:32.735893 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45"] Nov 03 22:47:32.742762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:32.742737 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:32.771687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:32.768042 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45"] Nov 03 22:47:32.809605 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:32.809569 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:47:32.809839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:32.809693 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:47:32.916719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:32.912665 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:32.916719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:32.912851 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgj7n\" (UniqueName: \"kubernetes.io/projected/853ed3df-7e6f-4457-9230-d141d3cccf85-kube-api-access-fgj7n\") pod \"maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:32.916719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:32.912920 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:33.018598 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:33.018555 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgj7n\" (UniqueName: \"kubernetes.io/projected/853ed3df-7e6f-4457-9230-d141d3cccf85-kube-api-access-fgj7n\") pod \"maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:33.018988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:33.018962 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:33.019387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:33.019362 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:33.021261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:33.021236 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:33.091582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:33.091546 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:33.096112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:33.096081 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgj7n\" (UniqueName: \"kubernetes.io/projected/853ed3df-7e6f-4457-9230-d141d3cccf85-kube-api-access-fgj7n\") pod \"maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:33.360223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:33.360191 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:33.774920 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:47:33.774887 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853ed3df_7e6f_4457_9230_d141d3cccf85.slice/crio-84a773d7d13f22abfc0bcc03a6af934dbbbe908079ffb1c10c0277abf2e29104 WatchSource:0}: Error finding container 84a773d7d13f22abfc0bcc03a6af934dbbbe908079ffb1c10c0277abf2e29104: Status 404 returned error can't find the container with id 84a773d7d13f22abfc0bcc03a6af934dbbbe908079ffb1c10c0277abf2e29104 Nov 03 22:47:33.779765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:33.779740 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45"] Nov 03 22:47:33.916230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:33.916184 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:34.302116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:34.302085 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:34.489573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:34.489542 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:34.677994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:34.677951 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:34.742036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:34.741955 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" event={"ID":"853ed3df-7e6f-4457-9230-d141d3cccf85","Type":"ContainerStarted","Data":"4dd0c6d6d097d37729707e8fe66d316a73e96cb2c1affbf1e5272ff9e95f9d94"} Nov 03 22:47:34.742036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:34.742012 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" event={"ID":"853ed3df-7e6f-4457-9230-d141d3cccf85","Type":"ContainerStarted","Data":"84a773d7d13f22abfc0bcc03a6af934dbbbe908079ffb1c10c0277abf2e29104"} Nov 03 22:47:34.895668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:34.895376 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" podStartSLOduration=2.895356525 podStartE2EDuration="2.895356525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:34.866217005 +0000 UTC m=+2411.846950659" watchObservedRunningTime="2026-11-03 22:47:34.895356525 +0000 UTC m=+2411.876090180" Nov 03 22:47:34.991047 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:34.991004 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk"] Nov 03 22:47:34.996186 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:34.996093 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3378fd-qnkpk"] Nov 03 22:47:35.136682 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:35.136636 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c51ea65-3567-4f09-a0be-1187c75d33ac" path="/var/lib/kubelet/pods/7c51ea65-3567-4f09-a0be-1187c75d33ac/volumes" Nov 03 22:47:35.358484 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:35.358327 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:35.563743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:35.563695 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:35.751370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:35.751336 4298 generic.go:356] "Generic (PLEG): container finished" podID="853ed3df-7e6f-4457-9230-d141d3cccf85" containerID="4dd0c6d6d097d37729707e8fe66d316a73e96cb2c1affbf1e5272ff9e95f9d94" exitCode=0 Nov 03 22:47:35.751548 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:35.751424 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" event={"ID":"853ed3df-7e6f-4457-9230-d141d3cccf85","Type":"ContainerDied","Data":"4dd0c6d6d097d37729707e8fe66d316a73e96cb2c1affbf1e5272ff9e95f9d94"} Nov 03 22:47:35.788865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:35.788799 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:35.987466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:35.987434 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:36.154482 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:36.154350 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj"] Nov 03 22:47:36.159496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:36.159471 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6a6b7f-wgrfj"] Nov 03 22:47:36.169663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:36.169624 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:36.918222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:36.918196 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:37.014472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.014399 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/853ed3df-7e6f-4457-9230-d141d3cccf85-kube-api-access-fgj7n\" (UniqueName: \"kubernetes.io/projected/853ed3df-7e6f-4457-9230-d141d3cccf85-kube-api-access-fgj7n\") pod \"853ed3df-7e6f-4457-9230-d141d3cccf85\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " Nov 03 22:47:37.014881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.014509 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-openshift-ca\") pod \"853ed3df-7e6f-4457-9230-d141d3cccf85\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " Nov 03 22:47:37.014881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.014750 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-xks-ca\") pod \"853ed3df-7e6f-4457-9230-d141d3cccf85\" (UID: \"853ed3df-7e6f-4457-9230-d141d3cccf85\") " Nov 03 22:47:37.014881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.014825 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-openshift-ca" pod "853ed3df-7e6f-4457-9230-d141d3cccf85" (UID: "853ed3df-7e6f-4457-9230-d141d3cccf85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:37.015225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.015205 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:37.016566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.016543 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853ed3df-7e6f-4457-9230-d141d3cccf85-kube-api-access-fgj7n" pod "853ed3df-7e6f-4457-9230-d141d3cccf85" (UID: "853ed3df-7e6f-4457-9230-d141d3cccf85"). InnerVolumeSpecName "kube-api-access-fgj7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:37.016801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.016774 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-xks-ca" pod "853ed3df-7e6f-4457-9230-d141d3cccf85" (UID: "853ed3df-7e6f-4457-9230-d141d3cccf85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:37.116692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.116660 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853ed3df-7e6f-4457-9230-d141d3cccf85-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:37.116692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.116688 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgj7n\" (UniqueName: \"kubernetes.io/projected/853ed3df-7e6f-4457-9230-d141d3cccf85-kube-api-access-fgj7n\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:37.143206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.143171 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a7c0e8-0477-4a3e-9b63-dbd93fb2b556" path="/var/lib/kubelet/pods/14a7c0e8-0477-4a3e-9b63-dbd93fb2b556/volumes" Nov 03 22:47:37.769154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.769128 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" Nov 03 22:47:37.769377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.769165 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45" event={"ID":"853ed3df-7e6f-4457-9230-d141d3cccf85","Type":"ContainerDied","Data":"84a773d7d13f22abfc0bcc03a6af934dbbbe908079ffb1c10c0277abf2e29104"} Nov 03 22:47:37.769377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:37.769209 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84a773d7d13f22abfc0bcc03a6af934dbbbe908079ffb1c10c0277abf2e29104" Nov 03 22:47:41.114704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:41.114673 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:41.370386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:41.370279 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5"] Nov 03 22:47:41.370563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:41.370500 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" containerID="cri-o://bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30" gracePeriod=30 Nov 03 22:47:41.390032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:41.390000 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8"] Nov 03 22:47:41.390231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:41.390210 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" podUID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" containerName="istio-proxy" containerID="cri-o://8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf" gracePeriod=30 Nov 03 22:47:41.395065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:41.395042 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48"] Nov 03 22:47:41.395228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:41.395209 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" podUID="c9295625-3631-4888-b199-02edcc9a4308" containerName="istio-proxy" containerID="cri-o://011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501" gracePeriod=30 Nov 03 22:47:41.680634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:41.680556 4298 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:47:41.680634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:41.680621 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:47:42.324441 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:42.324407 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:42.443631 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:42.443597 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq"] Nov 03 22:47:42.443843 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:42.443811 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" podUID="0c11a285-70b0-4657-9733-696a97ceabb9" containerName="istio-proxy" containerID="cri-o://32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc" gracePeriod=30 Nov 03 22:47:42.493891 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:42.493857 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn"] Nov 03 22:47:42.494084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:42.494065 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" podUID="165c2233-5f8d-426f-a423-de23cb1878ed" containerName="istio-proxy" containerID="cri-o://d6147b5e0fd745ab4f819c62b8f01ed758b2fc26ded146f40da5438e9d25f9cc" gracePeriod=30 Nov 03 22:47:42.513140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:42.513111 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:42.963500 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:42.963451 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:43.165795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:43.165763 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:43.356961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:43.356877 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:44.241204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:44.241171 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8"] Nov 03 22:47:44.349547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:44.349513 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt"] Nov 03 22:47:44.652717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:44.650393 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:44.879317 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:44.879290 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:46.458668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.454625 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:46.778001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.777970 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:47:46.780012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.779953 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:47:46.811919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.811894 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-podinfo\") pod \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " Nov 03 22:47:46.812069 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.811939 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9295625-3631-4888-b199-02edcc9a4308-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9295625-3631-4888-b199-02edcc9a4308-istiod-ca-cert\") pod \"c9295625-3631-4888-b199-02edcc9a4308\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " Nov 03 22:47:46.812069 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.811970 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9295625-3631-4888-b199-02edcc9a4308-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9295625-3631-4888-b199-02edcc9a4308-istio-podinfo\") pod \"c9295625-3631-4888-b199-02edcc9a4308\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " Nov 03 22:47:46.812069 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812021 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istiod-ca-cert\") pod \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " Nov 03 22:47:46.812069 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812066 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-credential-socket\") pod \"c9295625-3631-4888-b199-02edcc9a4308\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " Nov 03 22:47:46.812231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812112 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-envoy\") pod \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " Nov 03 22:47:46.812231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812165 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-socket\") pod \"c9295625-3631-4888-b199-02edcc9a4308\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " Nov 03 22:47:46.812231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812201 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-certs\") pod \"c9295625-3631-4888-b199-02edcc9a4308\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " Nov 03 22:47:46.812231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812227 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-data\") pod \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " Nov 03 22:47:46.812412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812283 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-kube-api-access-zmwgj\" (UniqueName: \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-kube-api-access-zmwgj\") pod \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " Nov 03 22:47:46.812412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812304 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-socket\") pod \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " Nov 03 22:47:46.812412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812305 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9295625-3631-4888-b199-02edcc9a4308-istiod-ca-cert" pod "c9295625-3631-4888-b199-02edcc9a4308" (UID: "c9295625-3631-4888-b199-02edcc9a4308"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:46.812412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812375 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-envoy\") pod \"c9295625-3631-4888-b199-02edcc9a4308\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " Nov 03 22:47:46.812625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812425 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-credential-socket\") pod \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " Nov 03 22:47:46.812625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812467 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-kube-api-access-kffh6\" (UniqueName: \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-kube-api-access-kffh6\") pod \"c9295625-3631-4888-b199-02edcc9a4308\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " Nov 03 22:47:46.812625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812504 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-certs\") pod \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " Nov 03 22:47:46.812625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812537 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-istio-token\" (UniqueName: \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-istio-token\") pod \"c9295625-3631-4888-b199-02edcc9a4308\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " Nov 03 22:47:46.812625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812561 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-data\") pod \"c9295625-3631-4888-b199-02edcc9a4308\" (UID: \"c9295625-3631-4888-b199-02edcc9a4308\") " Nov 03 22:47:46.812625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812625 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-token\" (UniqueName: \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-token\") pod \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\" (UID: \"2a07627c-ac0d-4c25-97ec-f60441a3a3b7\") " Nov 03 22:47:46.812980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812720 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-data" pod "2a07627c-ac0d-4c25-97ec-f60441a3a3b7" (UID: "2a07627c-ac0d-4c25-97ec-f60441a3a3b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.812980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812931 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-credential-socket" pod "2a07627c-ac0d-4c25-97ec-f60441a3a3b7" (UID: "2a07627c-ac0d-4c25-97ec-f60441a3a3b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.812980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.812949 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-certs" pod "c9295625-3631-4888-b199-02edcc9a4308" (UID: "c9295625-3631-4888-b199-02edcc9a4308"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.813273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.813244 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-credential-socket" pod "c9295625-3631-4888-b199-02edcc9a4308" (UID: "c9295625-3631-4888-b199-02edcc9a4308"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.813403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.813358 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.813476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.813380 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.813476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.813428 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.813476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.813443 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9295625-3631-4888-b199-02edcc9a4308-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.813692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.813494 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-socket" pod "c9295625-3631-4888-b199-02edcc9a4308" (UID: "c9295625-3631-4888-b199-02edcc9a4308"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.813692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.813661 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-certs" pod "2a07627c-ac0d-4c25-97ec-f60441a3a3b7" (UID: "2a07627c-ac0d-4c25-97ec-f60441a3a3b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.814071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.814043 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istiod-ca-cert" pod "2a07627c-ac0d-4c25-97ec-f60441a3a3b7" (UID: "2a07627c-ac0d-4c25-97ec-f60441a3a3b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:46.814455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.814351 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-podinfo" pod "2a07627c-ac0d-4c25-97ec-f60441a3a3b7" (UID: "2a07627c-ac0d-4c25-97ec-f60441a3a3b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:46.814455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.814419 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9295625-3631-4888-b199-02edcc9a4308-istio-podinfo" pod "c9295625-3631-4888-b199-02edcc9a4308" (UID: "c9295625-3631-4888-b199-02edcc9a4308"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:46.814683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.814619 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-socket" pod "2a07627c-ac0d-4c25-97ec-f60441a3a3b7" (UID: "2a07627c-ac0d-4c25-97ec-f60441a3a3b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.814908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.814872 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-data" pod "c9295625-3631-4888-b199-02edcc9a4308" (UID: "c9295625-3631-4888-b199-02edcc9a4308"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.815237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.815213 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-token" pod "2a07627c-ac0d-4c25-97ec-f60441a3a3b7" (UID: "2a07627c-ac0d-4c25-97ec-f60441a3a3b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.815452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.815435 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-kube-api-access-zmwgj" pod "2a07627c-ac0d-4c25-97ec-f60441a3a3b7" (UID: "2a07627c-ac0d-4c25-97ec-f60441a3a3b7"). InnerVolumeSpecName "kube-api-access-zmwgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.815659 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.815620 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-envoy" pod "2a07627c-ac0d-4c25-97ec-f60441a3a3b7" (UID: "2a07627c-ac0d-4c25-97ec-f60441a3a3b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.816005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.815985 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-kube-api-access-kffh6" pod "c9295625-3631-4888-b199-02edcc9a4308" (UID: "c9295625-3631-4888-b199-02edcc9a4308"). InnerVolumeSpecName "kube-api-access-kffh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.816210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.816195 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:47:46.816291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.816275 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-istio-token" pod "c9295625-3631-4888-b199-02edcc9a4308" (UID: "c9295625-3631-4888-b199-02edcc9a4308"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.816545 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.816531 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-envoy" pod "c9295625-3631-4888-b199-02edcc9a4308" (UID: "c9295625-3631-4888-b199-02edcc9a4308"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.857707 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.857672 4298 generic.go:356] "Generic (PLEG): container finished" podID="c9295625-3631-4888-b199-02edcc9a4308" containerID="011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501" exitCode=0 Nov 03 22:47:46.857910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.857736 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" Nov 03 22:47:46.857910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.857729 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" event={"ID":"c9295625-3631-4888-b199-02edcc9a4308","Type":"ContainerDied","Data":"011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501"} Nov 03 22:47:46.857910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.857865 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48" event={"ID":"c9295625-3631-4888-b199-02edcc9a4308","Type":"ContainerDied","Data":"e840e930921362e0c83df5e4dbd2fae4754f368bc9d3cd933a147fc862c02e02"} Nov 03 22:47:46.857910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.857895 4298 scope.go:122] "RemoveContainer" containerID="011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501" Nov 03 22:47:46.859270 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.859247 4298 generic.go:356] "Generic (PLEG): container finished" podID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerID="bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30" exitCode=0 Nov 03 22:47:46.859364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.859284 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" Nov 03 22:47:46.859364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.859324 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" event={"ID":"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851","Type":"ContainerDied","Data":"bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30"} Nov 03 22:47:46.859462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.859368 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5" event={"ID":"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851","Type":"ContainerDied","Data":"17e9681f4410eee754fa0a8f190f114feb31a60ac76ca04b619f28dad8990447"} Nov 03 22:47:46.860916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.860892 4298 generic.go:356] "Generic (PLEG): container finished" podID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" containerID="8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf" exitCode=0 Nov 03 22:47:46.861032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.860929 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" Nov 03 22:47:46.861032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.860980 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" event={"ID":"2a07627c-ac0d-4c25-97ec-f60441a3a3b7","Type":"ContainerDied","Data":"8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf"} Nov 03 22:47:46.861156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.861029 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8" event={"ID":"2a07627c-ac0d-4c25-97ec-f60441a3a3b7","Type":"ContainerDied","Data":"b2f2b20d0fff6590cd96ee41ae152bf112e7685b7a5f22d40a83ac9e9b26709d"} Nov 03 22:47:46.870205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.870187 4298 scope.go:122] "RemoveContainer" containerID="011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501" Nov 03 22:47:46.870540 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:47:46.870516 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501\": container with ID starting with 011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501 not found: ID does not exist" containerID="011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501" Nov 03 22:47:46.870606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.870554 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501"} err="failed to get container status \"011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501\": rpc error: code = NotFound desc = could not find container \"011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501\": container with ID starting with 011842a65725f503d7cb11bcc6a250afa67a402e102f62475271cc1e97f9e501 not found: ID does not exist" Nov 03 22:47:46.870606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.870581 4298 scope.go:122] "RemoveContainer" containerID="bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30" Nov 03 22:47:46.880473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.880453 4298 scope.go:122] "RemoveContainer" containerID="bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30" Nov 03 22:47:46.880712 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:47:46.880694 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30\": container with ID starting with bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30 not found: ID does not exist" containerID="bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30" Nov 03 22:47:46.880768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.880720 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30"} err="failed to get container status \"bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30\": rpc error: code = NotFound desc = could not find container \"bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30\": container with ID starting with bda8150448788d650b6ca8e8b23afc6eef738293d684915055922203b7041f30 not found: ID does not exist" Nov 03 22:47:46.880768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.880738 4298 scope.go:122] "RemoveContainer" containerID="8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf" Nov 03 22:47:46.890899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.890878 4298 scope.go:122] "RemoveContainer" containerID="8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf" Nov 03 22:47:46.891236 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:47:46.891210 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf\": container with ID starting with 8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf not found: ID does not exist" containerID="8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf" Nov 03 22:47:46.891337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.891243 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf"} err="failed to get container status \"8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf\": rpc error: code = NotFound desc = could not find container \"8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf\": container with ID starting with 8c15efa56d05f2902edbdbaaa054da3ad643c43eb1877ff77dbcf1289524c2bf not found: ID does not exist" Nov 03 22:47:46.913976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.913940 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istiod-ca-cert\") pod \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " Nov 03 22:47:46.914158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914129 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-credential-socket\") pod \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " Nov 03 22:47:46.914243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914228 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-data\") pod \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " Nov 03 22:47:46.914306 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914275 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-socket\") pod \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " Nov 03 22:47:46.914368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914340 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-envoy\") pod \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " Nov 03 22:47:46.914433 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914410 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-certs\") pod \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " Nov 03 22:47:46.914493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914447 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-token\" (UniqueName: \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-token\") pod \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " Nov 03 22:47:46.914493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914369 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istiod-ca-cert" pod "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" (UID: "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:46.914493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914448 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-credential-socket" pod "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" (UID: "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.914680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914480 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-kube-api-access-ktlfm\" (UniqueName: \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-kube-api-access-ktlfm\") pod \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " Nov 03 22:47:46.914680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914537 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-data" pod "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" (UID: "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.914680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914547 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-socket" pod "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" (UID: "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.914680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914564 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-podinfo\") pod \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\" (UID: \"80c2e0c5-cb35-4ce7-a07a-bae61ef1f851\") " Nov 03 22:47:46.914981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.914710 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-certs" pod "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" (UID: "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.915341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915322 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915346 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915361 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915376 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmwgj\" (UniqueName: \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-kube-api-access-zmwgj\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915391 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915405 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915419 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915433 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kffh6\" (UniqueName: \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-kube-api-access-kffh6\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915449 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915464 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915479 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915493 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9295625-3631-4888-b199-02edcc9a4308-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915507 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915521 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915535 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915549 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915563 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9295625-3631-4888-b199-02edcc9a4308-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915578 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a07627c-ac0d-4c25-97ec-f60441a3a3b7-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.915921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.915594 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9295625-3631-4888-b199-02edcc9a4308-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.916841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.916814 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-podinfo" pod "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" (UID: "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:46.917061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.917038 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-token" pod "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" (UID: "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.917061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.917046 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-envoy" pod "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" (UID: "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.917255 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:46.917068 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-kube-api-access-ktlfm" pod "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" (UID: "80c2e0c5-cb35-4ce7-a07a-bae61ef1f851"). InnerVolumeSpecName "kube-api-access-ktlfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:47.016791 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.016708 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.016791 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.016736 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.016791 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.016747 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktlfm\" (UniqueName: \"kubernetes.io/projected/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-kube-api-access-ktlfm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.016791 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.016756 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.165317 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.165284 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8"] Nov 03 22:47:47.168720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.168682 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-4vcw8"] Nov 03 22:47:47.322155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.322070 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48"] Nov 03 22:47:47.353022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.352988 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-kkh48"] Nov 03 22:47:47.580156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.580075 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5"] Nov 03 22:47:47.586503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.586469 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-mlpk5"] Nov 03 22:47:47.594752 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:47:47.594720 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:47:47.864531 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.864453 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:47:47.877974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.877941 4298 generic.go:356] "Generic (PLEG): container finished" podID="0c11a285-70b0-4657-9733-696a97ceabb9" containerID="32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc" exitCode=0 Nov 03 22:47:47.878128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.877987 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" Nov 03 22:47:47.878128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.878017 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" event={"ID":"0c11a285-70b0-4657-9733-696a97ceabb9","Type":"ContainerDied","Data":"32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc"} Nov 03 22:47:47.878128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.878092 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq" event={"ID":"0c11a285-70b0-4657-9733-696a97ceabb9","Type":"ContainerDied","Data":"64b12eaeca746318c0a0e6ff049dde886c9fe8bca079e26ecbcf07da6ee63676"} Nov 03 22:47:47.878128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.878106 4298 scope.go:122] "RemoveContainer" containerID="32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc" Nov 03 22:47:47.879809 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.879785 4298 generic.go:356] "Generic (PLEG): container finished" podID="165c2233-5f8d-426f-a423-de23cb1878ed" containerID="d6147b5e0fd745ab4f819c62b8f01ed758b2fc26ded146f40da5438e9d25f9cc" exitCode=0 Nov 03 22:47:47.879915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.879863 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" event={"ID":"165c2233-5f8d-426f-a423-de23cb1878ed","Type":"ContainerDied","Data":"d6147b5e0fd745ab4f819c62b8f01ed758b2fc26ded146f40da5438e9d25f9cc"} Nov 03 22:47:47.889705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.889683 4298 scope.go:122] "RemoveContainer" containerID="32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc" Nov 03 22:47:47.890721 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:47:47.889984 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc\": container with ID starting with 32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc not found: ID does not exist" containerID="32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc" Nov 03 22:47:47.890721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.890014 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc"} err="failed to get container status \"32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc\": rpc error: code = NotFound desc = could not find container \"32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc\": container with ID starting with 32f4e6e73efddab7ab0e57035701d03851bf731b2f8bae28143321f6433592bc not found: ID does not exist" Nov 03 22:47:47.904144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.904122 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:47:47.956474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956444 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-certs\") pod \"0c11a285-70b0-4657-9733-696a97ceabb9\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " Nov 03 22:47:47.956697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956481 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-kube-api-access-9kt86\" (UniqueName: \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-kube-api-access-9kt86\") pod \"165c2233-5f8d-426f-a423-de23cb1878ed\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " Nov 03 22:47:47.956697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956507 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-data\") pod \"165c2233-5f8d-426f-a423-de23cb1878ed\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " Nov 03 22:47:47.956697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956529 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-credential-socket\") pod \"0c11a285-70b0-4657-9733-696a97ceabb9\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " Nov 03 22:47:47.956697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956616 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-envoy\") pod \"165c2233-5f8d-426f-a423-de23cb1878ed\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " Nov 03 22:47:47.956697 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956681 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-kube-api-access-85qvt\" (UniqueName: \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-kube-api-access-85qvt\") pod \"0c11a285-70b0-4657-9733-696a97ceabb9\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " Nov 03 22:47:47.956998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956741 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c11a285-70b0-4657-9733-696a97ceabb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c11a285-70b0-4657-9733-696a97ceabb9-istio-podinfo\") pod \"0c11a285-70b0-4657-9733-696a97ceabb9\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " Nov 03 22:47:47.956998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956776 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-socket\") pod \"0c11a285-70b0-4657-9733-696a97ceabb9\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " Nov 03 22:47:47.956998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956800 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-istio-token\" (UniqueName: \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-istio-token\") pod \"0c11a285-70b0-4657-9733-696a97ceabb9\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " Nov 03 22:47:47.956998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956865 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-data\") pod \"0c11a285-70b0-4657-9733-696a97ceabb9\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " Nov 03 22:47:47.956998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956902 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-credential-socket\") pod \"165c2233-5f8d-426f-a423-de23cb1878ed\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " Nov 03 22:47:47.957281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.957003 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/165c2233-5f8d-426f-a423-de23cb1878ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165c2233-5f8d-426f-a423-de23cb1878ed-istio-podinfo\") pod \"165c2233-5f8d-426f-a423-de23cb1878ed\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " Nov 03 22:47:47.957281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.957034 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-istio-token\" (UniqueName: \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-istio-token\") pod \"165c2233-5f8d-426f-a423-de23cb1878ed\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " Nov 03 22:47:47.957281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.957065 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-certs\") pod \"165c2233-5f8d-426f-a423-de23cb1878ed\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " Nov 03 22:47:47.957281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.957100 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-socket\") pod \"165c2233-5f8d-426f-a423-de23cb1878ed\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " Nov 03 22:47:47.957281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.957131 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c11a285-70b0-4657-9733-696a97ceabb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c11a285-70b0-4657-9733-696a97ceabb9-istiod-ca-cert\") pod \"0c11a285-70b0-4657-9733-696a97ceabb9\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " Nov 03 22:47:47.957281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956794 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-certs" pod "0c11a285-70b0-4657-9733-696a97ceabb9" (UID: "0c11a285-70b0-4657-9733-696a97ceabb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:47.957281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956833 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-data" pod "165c2233-5f8d-426f-a423-de23cb1878ed" (UID: "165c2233-5f8d-426f-a423-de23cb1878ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:47.957281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.956898 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-credential-socket" pod "0c11a285-70b0-4657-9733-696a97ceabb9" (UID: "0c11a285-70b0-4657-9733-696a97ceabb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:47.957281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.957084 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-socket" pod "0c11a285-70b0-4657-9733-696a97ceabb9" (UID: "0c11a285-70b0-4657-9733-696a97ceabb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:47.957281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.957243 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-envoy\") pod \"0c11a285-70b0-4657-9733-696a97ceabb9\" (UID: \"0c11a285-70b0-4657-9733-696a97ceabb9\") " Nov 03 22:47:47.957281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.957257 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-credential-socket" pod "165c2233-5f8d-426f-a423-de23cb1878ed" (UID: "165c2233-5f8d-426f-a423-de23cb1878ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:47.957865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.957415 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-data" pod "0c11a285-70b0-4657-9733-696a97ceabb9" (UID: "0c11a285-70b0-4657-9733-696a97ceabb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:47.957865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.957456 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-certs" pod "165c2233-5f8d-426f-a423-de23cb1878ed" (UID: "165c2233-5f8d-426f-a423-de23cb1878ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:47.957865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.957616 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-socket" pod "165c2233-5f8d-426f-a423-de23cb1878ed" (UID: "165c2233-5f8d-426f-a423-de23cb1878ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:47.959256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.959232 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c11a285-70b0-4657-9733-696a97ceabb9-istiod-ca-cert" pod "0c11a285-70b0-4657-9733-696a97ceabb9" (UID: "0c11a285-70b0-4657-9733-696a97ceabb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:47.959416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.959386 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165c2233-5f8d-426f-a423-de23cb1878ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165c2233-5f8d-426f-a423-de23cb1878ed-istiod-ca-cert\") pod \"165c2233-5f8d-426f-a423-de23cb1878ed\" (UID: \"165c2233-5f8d-426f-a423-de23cb1878ed\") " Nov 03 22:47:47.959819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.959760 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-kube-api-access-9kt86" pod "165c2233-5f8d-426f-a423-de23cb1878ed" (UID: "165c2233-5f8d-426f-a423-de23cb1878ed"). InnerVolumeSpecName "kube-api-access-9kt86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:47.959819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.959784 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/165c2233-5f8d-426f-a423-de23cb1878ed-istio-podinfo" pod "165c2233-5f8d-426f-a423-de23cb1878ed" (UID: "165c2233-5f8d-426f-a423-de23cb1878ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:47.960024 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.959885 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-istio-token" pod "165c2233-5f8d-426f-a423-de23cb1878ed" (UID: "165c2233-5f8d-426f-a423-de23cb1878ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:47.960190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960168 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.960190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960188 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kt86\" (UniqueName: \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-kube-api-access-9kt86\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.960373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960199 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.960373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960208 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.960373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960216 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.960373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960224 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.960373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960217 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165c2233-5f8d-426f-a423-de23cb1878ed-istiod-ca-cert" pod "165c2233-5f8d-426f-a423-de23cb1878ed" (UID: "165c2233-5f8d-426f-a423-de23cb1878ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:47.960373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960232 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.960373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960241 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165c2233-5f8d-426f-a423-de23cb1878ed-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.960373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960249 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165c2233-5f8d-426f-a423-de23cb1878ed-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.960373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960257 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.960373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960265 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.960373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960274 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c11a285-70b0-4657-9733-696a97ceabb9-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.961038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960899 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c11a285-70b0-4657-9733-696a97ceabb9-istio-podinfo" pod "0c11a285-70b0-4657-9733-696a97ceabb9" (UID: "0c11a285-70b0-4657-9733-696a97ceabb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:47.961038 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.960950 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-kube-api-access-85qvt" pod "0c11a285-70b0-4657-9733-696a97ceabb9" (UID: "0c11a285-70b0-4657-9733-696a97ceabb9"). InnerVolumeSpecName "kube-api-access-85qvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:47.961261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.961240 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-envoy" pod "165c2233-5f8d-426f-a423-de23cb1878ed" (UID: "165c2233-5f8d-426f-a423-de23cb1878ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:47.961679 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.961655 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-envoy" pod "0c11a285-70b0-4657-9733-696a97ceabb9" (UID: "0c11a285-70b0-4657-9733-696a97ceabb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:47.962387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:47.962367 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-istio-token" pod "0c11a285-70b0-4657-9733-696a97ceabb9" (UID: "0c11a285-70b0-4657-9733-696a97ceabb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:48.062116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:48.062079 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165c2233-5f8d-426f-a423-de23cb1878ed-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.062116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:48.062115 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85qvt\" (UniqueName: \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-kube-api-access-85qvt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.062116 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:48.062126 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c11a285-70b0-4657-9733-696a97ceabb9-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.062422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:48.062136 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c11a285-70b0-4657-9733-696a97ceabb9-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.062422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:48.062145 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c11a285-70b0-4657-9733-696a97ceabb9-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.062422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:48.062154 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165c2233-5f8d-426f-a423-de23cb1878ed-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.313827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:48.313794 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq"] Nov 03 22:47:48.318391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:48.318363 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-nsrgq"] Nov 03 22:47:48.910319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:48.910279 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" event={"ID":"165c2233-5f8d-426f-a423-de23cb1878ed","Type":"ContainerDied","Data":"1bbc85238a9f5bf054b3eec4ea1a20fcb478cc4a83a8ca96d1a637cdc733f530"} Nov 03 22:47:48.910859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:48.910355 4298 scope.go:122] "RemoveContainer" containerID="d6147b5e0fd745ab4f819c62b8f01ed758b2fc26ded146f40da5438e9d25f9cc" Nov 03 22:47:48.910859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:48.910466 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn" Nov 03 22:47:49.162724 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.162588 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c11a285-70b0-4657-9733-696a97ceabb9" path="/var/lib/kubelet/pods/0c11a285-70b0-4657-9733-696a97ceabb9/volumes" Nov 03 22:47:49.167525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.167485 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" path="/var/lib/kubelet/pods/2a07627c-ac0d-4c25-97ec-f60441a3a3b7/volumes" Nov 03 22:47:49.168268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.168246 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" path="/var/lib/kubelet/pods/80c2e0c5-cb35-4ce7-a07a-bae61ef1f851/volumes" Nov 03 22:47:49.172170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.170738 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9295625-3631-4888-b199-02edcc9a4308" path="/var/lib/kubelet/pods/c9295625-3631-4888-b199-02edcc9a4308/volumes" Nov 03 22:47:49.180925 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.180851 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn"] Nov 03 22:47:49.188797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.188706 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-8vqcn"] Nov 03 22:47:49.248232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.243771 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" podUID="bb6eb1e2-b676-460c-85cc-a82b1f524cb5" containerName="payload-processing" containerID="cri-o://3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50" gracePeriod=25 Nov 03 22:47:49.350072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.349845 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" podUID="2741d23e-17ab-4c6a-8d88-1250c1a3db92" containerName="payload-pre-processing" containerID="cri-o://7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b" gracePeriod=25 Nov 03 22:47:49.587986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.587959 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:47:49.690101 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.690075 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-kube-api-access-m2hlt\" (UniqueName: \"kubernetes.io/projected/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-kube-api-access-m2hlt\") pod \"bb6eb1e2-b676-460c-85cc-a82b1f524cb5\" (UID: \"bb6eb1e2-b676-460c-85cc-a82b1f524cb5\") " Nov 03 22:47:49.690252 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.690151 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-plugins-config-volume\") pod \"bb6eb1e2-b676-460c-85cc-a82b1f524cb5\" (UID: \"bb6eb1e2-b676-460c-85cc-a82b1f524cb5\") " Nov 03 22:47:49.690483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.690462 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-plugins-config-volume" pod "bb6eb1e2-b676-460c-85cc-a82b1f524cb5" (UID: "bb6eb1e2-b676-460c-85cc-a82b1f524cb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:49.690737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.690714 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:49.692193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.692174 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-kube-api-access-m2hlt" pod "bb6eb1e2-b676-460c-85cc-a82b1f524cb5" (UID: "bb6eb1e2-b676-460c-85cc-a82b1f524cb5"). InnerVolumeSpecName "kube-api-access-m2hlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:49.719811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.719783 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:47:49.791551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.791521 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2741d23e-17ab-4c6a-8d88-1250c1a3db92-kube-api-access-8vgx4\" (UniqueName: \"kubernetes.io/projected/2741d23e-17ab-4c6a-8d88-1250c1a3db92-kube-api-access-8vgx4\") pod \"2741d23e-17ab-4c6a-8d88-1250c1a3db92\" (UID: \"2741d23e-17ab-4c6a-8d88-1250c1a3db92\") " Nov 03 22:47:49.791810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.791792 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2741d23e-17ab-4c6a-8d88-1250c1a3db92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741d23e-17ab-4c6a-8d88-1250c1a3db92-plugins-config-volume\") pod \"2741d23e-17ab-4c6a-8d88-1250c1a3db92\" (UID: \"2741d23e-17ab-4c6a-8d88-1250c1a3db92\") " Nov 03 22:47:49.792160 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.792135 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2741d23e-17ab-4c6a-8d88-1250c1a3db92-plugins-config-volume" pod "2741d23e-17ab-4c6a-8d88-1250c1a3db92" (UID: "2741d23e-17ab-4c6a-8d88-1250c1a3db92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:49.792228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.792214 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2hlt\" (UniqueName: \"kubernetes.io/projected/bb6eb1e2-b676-460c-85cc-a82b1f524cb5-kube-api-access-m2hlt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:49.793631 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.793598 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2741d23e-17ab-4c6a-8d88-1250c1a3db92-kube-api-access-8vgx4" pod "2741d23e-17ab-4c6a-8d88-1250c1a3db92" (UID: "2741d23e-17ab-4c6a-8d88-1250c1a3db92"). InnerVolumeSpecName "kube-api-access-8vgx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:49.893575 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.893538 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741d23e-17ab-4c6a-8d88-1250c1a3db92-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:49.893773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.893614 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vgx4\" (UniqueName: \"kubernetes.io/projected/2741d23e-17ab-4c6a-8d88-1250c1a3db92-kube-api-access-8vgx4\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:49.953591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.953550 4298 generic.go:356] "Generic (PLEG): container finished" podID="2741d23e-17ab-4c6a-8d88-1250c1a3db92" containerID="7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b" exitCode=0 Nov 03 22:47:49.954053 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.953697 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" event={"ID":"2741d23e-17ab-4c6a-8d88-1250c1a3db92","Type":"ContainerDied","Data":"7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b"} Nov 03 22:47:49.954053 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.953750 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" event={"ID":"2741d23e-17ab-4c6a-8d88-1250c1a3db92","Type":"ContainerDied","Data":"47308229684e3b2943ecd7a44525904247453886caef07a90ad211405e917fca"} Nov 03 22:47:49.954053 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.953781 4298 scope.go:122] "RemoveContainer" containerID="7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b" Nov 03 22:47:49.954053 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.953889 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt" Nov 03 22:47:49.958624 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.958595 4298 generic.go:356] "Generic (PLEG): container finished" podID="bb6eb1e2-b676-460c-85cc-a82b1f524cb5" containerID="3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50" exitCode=0 Nov 03 22:47:49.958826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.958701 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" event={"ID":"bb6eb1e2-b676-460c-85cc-a82b1f524cb5","Type":"ContainerDied","Data":"3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50"} Nov 03 22:47:49.958826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.958751 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" Nov 03 22:47:49.958826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.958760 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8" event={"ID":"bb6eb1e2-b676-460c-85cc-a82b1f524cb5","Type":"ContainerDied","Data":"9ceef41826eafc28f4c759e252800aa6565e25673d7331b6ddfb2da2a1efaf8d"} Nov 03 22:47:49.969690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.969636 4298 scope.go:122] "RemoveContainer" containerID="7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b" Nov 03 22:47:49.970061 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:47:49.970029 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b\": container with ID starting with 7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b not found: ID does not exist" containerID="7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b" Nov 03 22:47:49.970166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.970073 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b"} err="failed to get container status \"7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b\": rpc error: code = NotFound desc = could not find container \"7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b\": container with ID starting with 7c6a0d00b093c6502cd7c4b50db94ea0e414daae85909986db9ccf6d2cca240b not found: ID does not exist" Nov 03 22:47:49.970166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.970094 4298 scope.go:122] "RemoveContainer" containerID="3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50" Nov 03 22:47:49.980604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.980583 4298 scope.go:122] "RemoveContainer" containerID="3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50" Nov 03 22:47:49.980955 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:47:49.980929 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50\": container with ID starting with 3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50 not found: ID does not exist" containerID="3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50" Nov 03 22:47:49.981057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:49.980960 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50"} err="failed to get container status \"3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50\": rpc error: code = NotFound desc = could not find container \"3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50\": container with ID starting with 3dc2155caeb10702a870be46e85da8b8d9593d5bd3bf553a19e2ee0e0c2d4f50 not found: ID does not exist" Nov 03 22:47:50.123361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:50.123331 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8"] Nov 03 22:47:50.151400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:50.151364 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f881e3-5b4f6d65d6-9hzn8"] Nov 03 22:47:50.251856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:50.251754 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt"] Nov 03 22:47:50.257693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:50.257662 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f881e3-dd5dcbdc-gqxpt"] Nov 03 22:47:51.156389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:51.156346 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165c2233-5f8d-426f-a423-de23cb1878ed" path="/var/lib/kubelet/pods/165c2233-5f8d-426f-a423-de23cb1878ed/volumes" Nov 03 22:47:51.156833 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:51.156817 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2741d23e-17ab-4c6a-8d88-1250c1a3db92" path="/var/lib/kubelet/pods/2741d23e-17ab-4c6a-8d88-1250c1a3db92/volumes" Nov 03 22:47:51.157220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:51.157207 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6eb1e2-b676-460c-85cc-a82b1f524cb5" path="/var/lib/kubelet/pods/bb6eb1e2-b676-460c-85cc-a82b1f524cb5/volumes" Nov 03 22:47:51.426172 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:51.426079 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:55.012828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:55.012747 4298 generic.go:356] "Generic (PLEG): container finished" podID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerID="25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294" exitCode=0 Nov 03 22:47:55.012828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:55.012816 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" event={"ID":"36642f0e-de5f-4793-9398-0965f2d8ea70","Type":"ContainerDied","Data":"25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294"} Nov 03 22:47:55.013373 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:55.013298 4298 scope.go:122] "RemoveContainer" containerID="25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294" Nov 03 22:47:55.015113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:55.015087 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:47:55.015181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:55.015159 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:47:55.015220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:55.015194 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:47:56.048667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:56.045683 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" event={"ID":"36642f0e-de5f-4793-9398-0965f2d8ea70","Type":"ContainerStarted","Data":"44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5"} Nov 03 22:47:56.235625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:56.233560 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:56.478433 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:56.478396 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:47:56.478702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:56.478580 4298 patch_prober.go:29] interesting pod/e2e-api-b-295682-openshift-default-7446766979-95n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:47:56.478702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:56.478670 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:47:57.477844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:57.477807 4298 patch_prober.go:29] interesting pod/e2e-api-b-295682-openshift-default-7446766979-95n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:47:57.478347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:57.477864 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:47:58.477964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:58.477918 4298 patch_prober.go:29] interesting pod/e2e-api-b-295682-openshift-default-7446766979-95n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:47:58.478434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:58.477985 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:47:59.477563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:59.477530 4298 patch_prober.go:29] interesting pod/e2e-api-b-295682-openshift-default-7446766979-95n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:47:59.477766 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:47:59.477581 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:48:00.478044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:00.478006 4298 patch_prober.go:29] interesting pod/e2e-api-b-295682-openshift-default-7446766979-95n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:48:00.478505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:00.478070 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:48:01.057996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:01.057940 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:01.478048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:01.478011 4298 patch_prober.go:29] interesting pod/e2e-api-b-295682-openshift-default-7446766979-95n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:48:01.478470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:01.478066 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:48:02.204858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:02.204825 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:48:02.483131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:02.483057 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:48:03.113865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:03.113839 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:48:05.876229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:05.876186 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:06.774558 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:06.774517 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:10.711764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:10.711727 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:15.526416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:15.526323 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:15.777528 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:15.777446 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:15.955863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:15.955830 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:16.988309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:16.988260 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:17.179283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:17.179251 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:17.372101 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:17.372023 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:17.552477 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:48:17.552442 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:48:20.465784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:20.465747 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:21.832302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:21.832266 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:22.179113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179038 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2"] Nov 03 22:48:22.179699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179676 4298 cpu_manager.go:414] "containerMap: removing container" podUID="2741d23e-17ab-4c6a-8d88-1250c1a3db92" containerName="payload-pre-processing" Nov 03 22:48:22.179699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179692 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2741d23e-17ab-4c6a-8d88-1250c1a3db92" containerName="payload-pre-processing" Nov 03 22:48:22.179699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179704 4298 cpu_manager.go:414] "containerMap: removing container" podUID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" containerName="istio-proxy" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179710 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" containerName="istio-proxy" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179722 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c9295625-3631-4888-b199-02edcc9a4308" containerName="istio-proxy" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179727 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9295625-3631-4888-b199-02edcc9a4308" containerName="istio-proxy" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179751 4298 cpu_manager.go:414] "containerMap: removing container" podUID="bb6eb1e2-b676-460c-85cc-a82b1f524cb5" containerName="payload-processing" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179756 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6eb1e2-b676-460c-85cc-a82b1f524cb5" containerName="payload-processing" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179766 4298 cpu_manager.go:414] "containerMap: removing container" podUID="0c11a285-70b0-4657-9733-696a97ceabb9" containerName="istio-proxy" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179771 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c11a285-70b0-4657-9733-696a97ceabb9" containerName="istio-proxy" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179780 4298 cpu_manager.go:414] "containerMap: removing container" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179784 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179789 4298 cpu_manager.go:414] "containerMap: removing container" podUID="165c2233-5f8d-426f-a423-de23cb1878ed" containerName="istio-proxy" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179794 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165c2233-5f8d-426f-a423-de23cb1878ed" containerName="istio-proxy" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179802 4298 cpu_manager.go:414] "containerMap: removing container" podUID="853ed3df-7e6f-4457-9230-d141d3cccf85" containerName="revoke-keys" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179806 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853ed3df-7e6f-4457-9230-d141d3cccf85" containerName="revoke-keys" Nov 03 22:48:22.179895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179899 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c11a285-70b0-4657-9733-696a97ceabb9" containerName="istio-proxy" Nov 03 22:48:22.180337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179907 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c2e0c5-cb35-4ce7-a07a-bae61ef1f851" containerName="istio-proxy" Nov 03 22:48:22.180337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179913 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="165c2233-5f8d-426f-a423-de23cb1878ed" containerName="istio-proxy" Nov 03 22:48:22.180337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179919 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="853ed3df-7e6f-4457-9230-d141d3cccf85" containerName="revoke-keys" Nov 03 22:48:22.180337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179929 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="2741d23e-17ab-4c6a-8d88-1250c1a3db92" containerName="payload-pre-processing" Nov 03 22:48:22.180337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179934 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a07627c-ac0d-4c25-97ec-f60441a3a3b7" containerName="istio-proxy" Nov 03 22:48:22.180337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179941 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9295625-3631-4888-b199-02edcc9a4308" containerName="istio-proxy" Nov 03 22:48:22.180337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.179953 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6eb1e2-b676-460c-85cc-a82b1f524cb5" containerName="payload-processing" Nov 03 22:48:22.183581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.183566 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:22.240466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.240437 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2"] Nov 03 22:48:22.284839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.284797 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:22.285061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.284843 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crr8q\" (UniqueName: \"kubernetes.io/projected/ceae44a9-cee1-4188-aac6-9e190b7b3e66-kube-api-access-crr8q\") pod \"maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:22.285061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.284882 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:48:22.285061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.284998 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:48:22.285254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.285182 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:22.386403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.386372 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:22.386403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.386405 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crr8q\" (UniqueName: \"kubernetes.io/projected/ceae44a9-cee1-4188-aac6-9e190b7b3e66-kube-api-access-crr8q\") pod \"maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:22.386693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.386668 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:22.387091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.387072 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:22.388933 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.388910 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:22.443096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.443023 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crr8q\" (UniqueName: \"kubernetes.io/projected/ceae44a9-cee1-4188-aac6-9e190b7b3e66-kube-api-access-crr8q\") pod \"maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:22.496895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.496861 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:22.867212 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:48:22.866557 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceae44a9_cee1_4188_aac6_9e190b7b3e66.slice/crio-9c340ca8bdc488d44a857a96e96c2a13bbd50fe3c41f3fa9302a1201ee016874 WatchSource:0}: Error finding container 9c340ca8bdc488d44a857a96e96c2a13bbd50fe3c41f3fa9302a1201ee016874: Status 404 returned error can't find the container with id 9c340ca8bdc488d44a857a96e96c2a13bbd50fe3c41f3fa9302a1201ee016874 Nov 03 22:48:22.870767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:22.870744 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2"] Nov 03 22:48:23.287206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:23.287161 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" event={"ID":"ceae44a9-cee1-4188-aac6-9e190b7b3e66","Type":"ContainerStarted","Data":"f2aafd3c0a98c4636e4752cdb18f78b7511609d974f0b23be1bdc02a6e905e89"} Nov 03 22:48:23.287382 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:23.287233 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" event={"ID":"ceae44a9-cee1-4188-aac6-9e190b7b3e66","Type":"ContainerStarted","Data":"9c340ca8bdc488d44a857a96e96c2a13bbd50fe3c41f3fa9302a1201ee016874"} Nov 03 22:48:23.401368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:23.401262 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" podStartSLOduration=1.401246265 podStartE2EDuration="1.401246265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:23.400099784 +0000 UTC m=+2460.380833438" watchObservedRunningTime="2026-11-03 22:48:23.401246265 +0000 UTC m=+2460.381979919" Nov 03 22:48:24.295686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:24.295622 4298 generic.go:356] "Generic (PLEG): container finished" podID="ceae44a9-cee1-4188-aac6-9e190b7b3e66" containerID="f2aafd3c0a98c4636e4752cdb18f78b7511609d974f0b23be1bdc02a6e905e89" exitCode=0 Nov 03 22:48:24.296062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:24.295692 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" event={"ID":"ceae44a9-cee1-4188-aac6-9e190b7b3e66","Type":"ContainerDied","Data":"f2aafd3c0a98c4636e4752cdb18f78b7511609d974f0b23be1bdc02a6e905e89"} Nov 03 22:48:25.468153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:25.468130 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:25.524533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:25.524503 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:25.529765 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:25.529733 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-openshift-ca\") pod \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " Nov 03 22:48:25.551405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:25.551351 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceae44a9-cee1-4188-aac6-9e190b7b3e66-kube-api-access-crr8q\" (UniqueName: \"kubernetes.io/projected/ceae44a9-cee1-4188-aac6-9e190b7b3e66-kube-api-access-crr8q\") pod \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " Nov 03 22:48:25.551557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:25.551524 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-xks-ca\") pod \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\" (UID: \"ceae44a9-cee1-4188-aac6-9e190b7b3e66\") " Nov 03 22:48:25.554102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:25.531223 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-openshift-ca" pod "ceae44a9-cee1-4188-aac6-9e190b7b3e66" (UID: "ceae44a9-cee1-4188-aac6-9e190b7b3e66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:25.557136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:25.557068 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceae44a9-cee1-4188-aac6-9e190b7b3e66-kube-api-access-crr8q" pod "ceae44a9-cee1-4188-aac6-9e190b7b3e66" (UID: "ceae44a9-cee1-4188-aac6-9e190b7b3e66"). InnerVolumeSpecName "kube-api-access-crr8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:25.558562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:25.558525 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-xks-ca" pod "ceae44a9-cee1-4188-aac6-9e190b7b3e66" (UID: "ceae44a9-cee1-4188-aac6-9e190b7b3e66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:25.653607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:25.653559 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:25.653607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:25.653603 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceae44a9-cee1-4188-aac6-9e190b7b3e66-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:25.653607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:25.653614 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crr8q\" (UniqueName: \"kubernetes.io/projected/ceae44a9-cee1-4188-aac6-9e190b7b3e66-kube-api-access-crr8q\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:26.311130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:26.311102 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" Nov 03 22:48:26.311309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:26.311141 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2" event={"ID":"ceae44a9-cee1-4188-aac6-9e190b7b3e66","Type":"ContainerDied","Data":"9c340ca8bdc488d44a857a96e96c2a13bbd50fe3c41f3fa9302a1201ee016874"} Nov 03 22:48:26.311309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:26.311196 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c340ca8bdc488d44a857a96e96c2a13bbd50fe3c41f3fa9302a1201ee016874" Nov 03 22:48:26.873777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:26.873741 4298 scope.go:122] "RemoveContainer" containerID="8b0ccd5ca8d9e32e28e87d54e2d6beb149134e8517a6754f682465e09da9f073" Nov 03 22:48:27.282984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:27.282952 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:30.285292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:30.284971 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:32.149320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.149281 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp"] Nov 03 22:48:32.151449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.151423 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ceae44a9-cee1-4188-aac6-9e190b7b3e66" containerName="revoke-keys" Nov 03 22:48:32.151610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.151596 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceae44a9-cee1-4188-aac6-9e190b7b3e66" containerName="revoke-keys" Nov 03 22:48:32.152155 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.152136 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceae44a9-cee1-4188-aac6-9e190b7b3e66" containerName="revoke-keys" Nov 03 22:48:32.157496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.157479 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:32.209710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.209683 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp"] Nov 03 22:48:32.243365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.243331 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:32.243542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.243396 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:32.243542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.243504 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42sm\" (UniqueName: \"kubernetes.io/projected/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-kube-api-access-x42sm\") pod \"maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:32.254522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.254498 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:48:32.254522 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.254516 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:48:32.344621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.344580 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:32.344857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.344636 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:32.344857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.344725 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42sm\" (UniqueName: \"kubernetes.io/projected/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-kube-api-access-x42sm\") pod \"maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:32.345375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.345344 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:32.346927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.346909 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:32.401776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.401709 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42sm\" (UniqueName: \"kubernetes.io/projected/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-kube-api-access-x42sm\") pod \"maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:32.472768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.472737 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:32.843689 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:48:32.843531 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03919c76_4d0f_43a7_b3aa_aa72a98e2e5c.slice/crio-e2c66ce85005dddabfd492203fe9b63f902166fc3e91dea15b0f76bb3c1639b5 WatchSource:0}: Error finding container e2c66ce85005dddabfd492203fe9b63f902166fc3e91dea15b0f76bb3c1639b5: Status 404 returned error can't find the container with id e2c66ce85005dddabfd492203fe9b63f902166fc3e91dea15b0f76bb3c1639b5 Nov 03 22:48:32.848452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:32.848403 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp"] Nov 03 22:48:33.365562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:33.365519 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" event={"ID":"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c","Type":"ContainerStarted","Data":"d0ae1c8d48bb4ae2beead722e9e27a74985486a1a9ad897e98c47f728818ee9e"} Nov 03 22:48:33.366020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:33.365582 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" event={"ID":"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c","Type":"ContainerStarted","Data":"e2c66ce85005dddabfd492203fe9b63f902166fc3e91dea15b0f76bb3c1639b5"} Nov 03 22:48:33.481685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:33.481615 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" podStartSLOduration=1.481600472 podStartE2EDuration="1.481600472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:33.479276685 +0000 UTC m=+2470.460010338" watchObservedRunningTime="2026-11-03 22:48:33.481600472 +0000 UTC m=+2470.462334124" Nov 03 22:48:34.150299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:34.150262 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v"] Nov 03 22:48:34.262733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:34.262701 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb"] Nov 03 22:48:34.374349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:34.374315 4298 generic.go:356] "Generic (PLEG): container finished" podID="03919c76-4d0f-43a7-b3aa-aa72a98e2e5c" containerID="d0ae1c8d48bb4ae2beead722e9e27a74985486a1a9ad897e98c47f728818ee9e" exitCode=0 Nov 03 22:48:34.374754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:34.374393 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" event={"ID":"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c","Type":"ContainerDied","Data":"d0ae1c8d48bb4ae2beead722e9e27a74985486a1a9ad897e98c47f728818ee9e"} Nov 03 22:48:34.557486 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:34.557457 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:34.747824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:34.747766 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:35.537098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:35.537063 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:35.546622 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:35.546586 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:35.705666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:35.705399 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-xks-ca\") pod \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " Nov 03 22:48:35.705666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:35.705590 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-openshift-ca\") pod \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " Nov 03 22:48:35.705939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:35.705698 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-kube-api-access-x42sm\" (UniqueName: \"kubernetes.io/projected/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-kube-api-access-x42sm\") pod \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\" (UID: \"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c\") " Nov 03 22:48:35.716450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:35.706834 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-openshift-ca" pod "03919c76-4d0f-43a7-b3aa-aa72a98e2e5c" (UID: "03919c76-4d0f-43a7-b3aa-aa72a98e2e5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:35.727365 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:35.718053 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-xks-ca" pod "03919c76-4d0f-43a7-b3aa-aa72a98e2e5c" (UID: "03919c76-4d0f-43a7-b3aa-aa72a98e2e5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:35.732843 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:35.732806 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-kube-api-access-x42sm" pod "03919c76-4d0f-43a7-b3aa-aa72a98e2e5c" (UID: "03919c76-4d0f-43a7-b3aa-aa72a98e2e5c"). InnerVolumeSpecName "kube-api-access-x42sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:35.808464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:35.808420 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:35.808464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:35.808463 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:35.808795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:35.808478 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42sm\" (UniqueName: \"kubernetes.io/projected/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c-kube-api-access-x42sm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:36.392140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:36.392111 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" Nov 03 22:48:36.392369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:36.392151 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp" event={"ID":"03919c76-4d0f-43a7-b3aa-aa72a98e2e5c","Type":"ContainerDied","Data":"e2c66ce85005dddabfd492203fe9b63f902166fc3e91dea15b0f76bb3c1639b5"} Nov 03 22:48:36.392369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:36.392198 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2c66ce85005dddabfd492203fe9b63f902166fc3e91dea15b0f76bb3c1639b5" Nov 03 22:48:39.150884 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.150831 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" podUID="f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07" containerName="payload-processing" containerID="cri-o://be61ee6775ca50352d51ff875a16f120090849d7d339038abe15a0e41232f2bc" gracePeriod=25 Nov 03 22:48:39.263393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.263355 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" podUID="df3bfd04-fff1-46d3-865e-b559ac337802" containerName="payload-pre-processing" containerID="cri-o://4ce5366c9ec168d5736c5076a60962a101d8a6c627ec8308a0bbee4e16e36b6d" gracePeriod=25 Nov 03 22:48:39.420337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.420241 4298 generic.go:356] "Generic (PLEG): container finished" podID="df3bfd04-fff1-46d3-865e-b559ac337802" containerID="4ce5366c9ec168d5736c5076a60962a101d8a6c627ec8308a0bbee4e16e36b6d" exitCode=0 Nov 03 22:48:39.420516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.420323 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" event={"ID":"df3bfd04-fff1-46d3-865e-b559ac337802","Type":"ContainerDied","Data":"4ce5366c9ec168d5736c5076a60962a101d8a6c627ec8308a0bbee4e16e36b6d"} Nov 03 22:48:39.422097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.422071 4298 generic.go:356] "Generic (PLEG): container finished" podID="f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07" containerID="be61ee6775ca50352d51ff875a16f120090849d7d339038abe15a0e41232f2bc" exitCode=0 Nov 03 22:48:39.422229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.422127 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" event={"ID":"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07","Type":"ContainerDied","Data":"be61ee6775ca50352d51ff875a16f120090849d7d339038abe15a0e41232f2bc"} Nov 03 22:48:39.422229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.422162 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" event={"ID":"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07","Type":"ContainerDied","Data":"b868cc72fc12bb7d0e577fcc62b7952827e2c72923f8bed4a7026001092fcc10"} Nov 03 22:48:39.422229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.422186 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b868cc72fc12bb7d0e577fcc62b7952827e2c72923f8bed4a7026001092fcc10" Nov 03 22:48:39.427022 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.427002 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:48:39.463355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.463327 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-kube-api-access-pt72w\" (UniqueName: \"kubernetes.io/projected/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-kube-api-access-pt72w\") pod \"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07\" (UID: \"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07\") " Nov 03 22:48:39.463546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.463456 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-plugins-config-volume\") pod \"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07\" (UID: \"f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07\") " Nov 03 22:48:39.464052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.464019 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-plugins-config-volume" pod "f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07" (UID: "f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:39.466088 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.466064 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-kube-api-access-pt72w" pod "f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07" (UID: "f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07"). InnerVolumeSpecName "kube-api-access-pt72w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:39.527513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.527491 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:48:39.564886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.564858 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3bfd04-fff1-46d3-865e-b559ac337802-kube-api-access-d52sf\" (UniqueName: \"kubernetes.io/projected/df3bfd04-fff1-46d3-865e-b559ac337802-kube-api-access-d52sf\") pod \"df3bfd04-fff1-46d3-865e-b559ac337802\" (UID: \"df3bfd04-fff1-46d3-865e-b559ac337802\") " Nov 03 22:48:39.565092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.565059 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df3bfd04-fff1-46d3-865e-b559ac337802-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3bfd04-fff1-46d3-865e-b559ac337802-plugins-config-volume\") pod \"df3bfd04-fff1-46d3-865e-b559ac337802\" (UID: \"df3bfd04-fff1-46d3-865e-b559ac337802\") " Nov 03 22:48:39.565409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.565386 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3bfd04-fff1-46d3-865e-b559ac337802-plugins-config-volume" pod "df3bfd04-fff1-46d3-865e-b559ac337802" (UID: "df3bfd04-fff1-46d3-865e-b559ac337802"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:39.565636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.565611 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:39.565762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.565675 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3bfd04-fff1-46d3-865e-b559ac337802-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:39.565762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.565692 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt72w\" (UniqueName: \"kubernetes.io/projected/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07-kube-api-access-pt72w\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:39.567166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.567143 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3bfd04-fff1-46d3-865e-b559ac337802-kube-api-access-d52sf" pod "df3bfd04-fff1-46d3-865e-b559ac337802" (UID: "df3bfd04-fff1-46d3-865e-b559ac337802"). InnerVolumeSpecName "kube-api-access-d52sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:39.666864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:39.666835 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d52sf\" (UniqueName: \"kubernetes.io/projected/df3bfd04-fff1-46d3-865e-b559ac337802-kube-api-access-d52sf\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:40.430681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:40.430579 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" Nov 03 22:48:40.430681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:40.430612 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb" event={"ID":"df3bfd04-fff1-46d3-865e-b559ac337802","Type":"ContainerDied","Data":"3b836bd716c2613293bc44d90eb1db23e93486e8623057536b9823d3c2020cf2"} Nov 03 22:48:40.431165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:40.430687 4298 scope.go:122] "RemoveContainer" containerID="4ce5366c9ec168d5736c5076a60962a101d8a6c627ec8308a0bbee4e16e36b6d" Nov 03 22:48:40.431165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:40.430836 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v" Nov 03 22:48:40.598839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:40.598805 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v"] Nov 03 22:48:40.648974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:40.648943 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b2fb1-845577f9db-g7v4v"] Nov 03 22:48:40.710067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:40.710037 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb"] Nov 03 22:48:40.714620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:40.714599 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b2fb1-6ff99d4887-4dvfb"] Nov 03 22:48:40.765409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:40.765374 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:41.135602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:41.135525 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df3bfd04-fff1-46d3-865e-b559ac337802" path="/var/lib/kubelet/pods/df3bfd04-fff1-46d3-865e-b559ac337802/volumes" Nov 03 22:48:41.135969 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:41.135955 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07" path="/var/lib/kubelet/pods/f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07/volumes" Nov 03 22:48:43.662287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:43.662252 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg"] Nov 03 22:48:43.772804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:43.772774 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m"] Nov 03 22:48:44.068991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:44.068955 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:44.260152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:44.260103 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:45.125153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:45.125084 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:46.737012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:46.736980 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:46.931807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:46.931777 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:47.107327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:47.107252 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:47.204029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:47.203997 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45"] Nov 03 22:48:47.260566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:47.260522 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f881e3-s5d45"] Nov 03 22:48:47.552410 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:48:47.552375 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:48:47.658681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:47.658637 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:48.662708 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:48.662667 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" podUID="e0ac7245-3eba-4cfe-bbff-58b56adf7b48" containerName="payload-processing" containerID="cri-o://7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877" gracePeriod=25 Nov 03 22:48:48.773129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:48.773067 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" podUID="78598387-4616-4d8f-9d4b-acb3f5331af6" containerName="payload-pre-processing" containerID="cri-o://e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d" gracePeriod=25 Nov 03 22:48:48.951084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:48.951060 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:48:49.018003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.017966 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-plugins-config-volume\") pod \"e0ac7245-3eba-4cfe-bbff-58b56adf7b48\" (UID: \"e0ac7245-3eba-4cfe-bbff-58b56adf7b48\") " Nov 03 22:48:49.018219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.018165 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-kube-api-access-svw8z\" (UniqueName: \"kubernetes.io/projected/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-kube-api-access-svw8z\") pod \"e0ac7245-3eba-4cfe-bbff-58b56adf7b48\" (UID: \"e0ac7245-3eba-4cfe-bbff-58b56adf7b48\") " Nov 03 22:48:49.019489 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.019463 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-plugins-config-volume" pod "e0ac7245-3eba-4cfe-bbff-58b56adf7b48" (UID: "e0ac7245-3eba-4cfe-bbff-58b56adf7b48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:49.021483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.021450 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-kube-api-access-svw8z" pod "e0ac7245-3eba-4cfe-bbff-58b56adf7b48" (UID: "e0ac7245-3eba-4cfe-bbff-58b56adf7b48"). InnerVolumeSpecName "kube-api-access-svw8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:49.055867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.055844 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:48:49.120104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.120071 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78598387-4616-4d8f-9d4b-acb3f5331af6-kube-api-access-kfk7t\" (UniqueName: \"kubernetes.io/projected/78598387-4616-4d8f-9d4b-acb3f5331af6-kube-api-access-kfk7t\") pod \"78598387-4616-4d8f-9d4b-acb3f5331af6\" (UID: \"78598387-4616-4d8f-9d4b-acb3f5331af6\") " Nov 03 22:48:49.120322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.120304 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78598387-4616-4d8f-9d4b-acb3f5331af6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78598387-4616-4d8f-9d4b-acb3f5331af6-plugins-config-volume\") pod \"78598387-4616-4d8f-9d4b-acb3f5331af6\" (UID: \"78598387-4616-4d8f-9d4b-acb3f5331af6\") " Nov 03 22:48:49.120769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.120743 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78598387-4616-4d8f-9d4b-acb3f5331af6-plugins-config-volume" pod "78598387-4616-4d8f-9d4b-acb3f5331af6" (UID: "78598387-4616-4d8f-9d4b-acb3f5331af6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:49.120927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.120911 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svw8z\" (UniqueName: \"kubernetes.io/projected/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-kube-api-access-svw8z\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:49.121011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.120928 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78598387-4616-4d8f-9d4b-acb3f5331af6-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:49.121011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.120937 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ac7245-3eba-4cfe-bbff-58b56adf7b48-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:49.122798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.122774 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78598387-4616-4d8f-9d4b-acb3f5331af6-kube-api-access-kfk7t" pod "78598387-4616-4d8f-9d4b-acb3f5331af6" (UID: "78598387-4616-4d8f-9d4b-acb3f5331af6"). InnerVolumeSpecName "kube-api-access-kfk7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:49.136922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.136897 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="853ed3df-7e6f-4457-9230-d141d3cccf85" path="/var/lib/kubelet/pods/853ed3df-7e6f-4457-9230-d141d3cccf85/volumes" Nov 03 22:48:49.222054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.222023 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfk7t\" (UniqueName: \"kubernetes.io/projected/78598387-4616-4d8f-9d4b-acb3f5331af6-kube-api-access-kfk7t\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:49.501425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.501335 4298 generic.go:356] "Generic (PLEG): container finished" podID="78598387-4616-4d8f-9d4b-acb3f5331af6" containerID="e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d" exitCode=0 Nov 03 22:48:49.501425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.501381 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" Nov 03 22:48:49.501637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.501427 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" event={"ID":"78598387-4616-4d8f-9d4b-acb3f5331af6","Type":"ContainerDied","Data":"e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d"} Nov 03 22:48:49.501637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.501480 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m" event={"ID":"78598387-4616-4d8f-9d4b-acb3f5331af6","Type":"ContainerDied","Data":"ef3e2f8ecea6372d0b0bc0ce88b7cb8741cb4c72ed3a6cd25c89a87449414e4a"} Nov 03 22:48:49.501637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.501507 4298 scope.go:122] "RemoveContainer" containerID="e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d" Nov 03 22:48:49.502735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.502710 4298 generic.go:356] "Generic (PLEG): container finished" podID="e0ac7245-3eba-4cfe-bbff-58b56adf7b48" containerID="7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877" exitCode=0 Nov 03 22:48:49.502873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.502760 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" Nov 03 22:48:49.502873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.502792 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" event={"ID":"e0ac7245-3eba-4cfe-bbff-58b56adf7b48","Type":"ContainerDied","Data":"7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877"} Nov 03 22:48:49.502873 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.502829 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg" event={"ID":"e0ac7245-3eba-4cfe-bbff-58b56adf7b48","Type":"ContainerDied","Data":"a55f31a663761bb83c959786ee74efa9d8ac0c6337f2e0a251bf8d60b37f8c5c"} Nov 03 22:48:49.511864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.511846 4298 scope.go:122] "RemoveContainer" containerID="e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d" Nov 03 22:48:49.512130 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:48:49.512107 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d\": container with ID starting with e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d not found: ID does not exist" containerID="e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d" Nov 03 22:48:49.512193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.512129 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d"} err="failed to get container status \"e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d\": rpc error: code = NotFound desc = could not find container \"e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d\": container with ID starting with e65c867c8178012bc9b05cd8335364bd98520d894a5c038edfcc70d5c7ff8f3d not found: ID does not exist" Nov 03 22:48:49.512193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.512145 4298 scope.go:122] "RemoveContainer" containerID="7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877" Nov 03 22:48:49.524555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.524525 4298 scope.go:122] "RemoveContainer" containerID="7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877" Nov 03 22:48:49.524843 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:48:49.524817 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877\": container with ID starting with 7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877 not found: ID does not exist" containerID="7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877" Nov 03 22:48:49.524938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.524850 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877"} err="failed to get container status \"7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877\": rpc error: code = NotFound desc = could not find container \"7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877\": container with ID starting with 7bc8ec83d3f02e575e16d564c702ce7c68a1e4cffcd4e078981c1c0e9dfb3877 not found: ID does not exist" Nov 03 22:48:49.669856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.669826 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg"] Nov 03 22:48:49.677290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.677264 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f168-fddcd4658-6szdg"] Nov 03 22:48:49.780410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.780318 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m"] Nov 03 22:48:49.786392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:49.786366 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f168-55959c7ff-qds5m"] Nov 03 22:48:51.135406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:51.135374 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78598387-4616-4d8f-9d4b-acb3f5331af6" path="/var/lib/kubelet/pods/78598387-4616-4d8f-9d4b-acb3f5331af6/volumes" Nov 03 22:48:51.135821 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:51.135807 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ac7245-3eba-4cfe-bbff-58b56adf7b48" path="/var/lib/kubelet/pods/e0ac7245-3eba-4cfe-bbff-58b56adf7b48/volumes" Nov 03 22:48:52.136828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:52.136791 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:53.452033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:53.452000 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:53.647663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:53.644421 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:53.675940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:53.675907 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h"] Nov 03 22:48:53.676142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:53.676120 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" containerID="cri-o://8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594" gracePeriod=30 Nov 03 22:48:53.855954 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:53.855758 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:54.274753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.274718 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r"] Nov 03 22:48:54.275356 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275342 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e0ac7245-3eba-4cfe-bbff-58b56adf7b48" containerName="payload-processing" Nov 03 22:48:54.275412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275358 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ac7245-3eba-4cfe-bbff-58b56adf7b48" containerName="payload-processing" Nov 03 22:48:54.275412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275379 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07" containerName="payload-processing" Nov 03 22:48:54.275412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275384 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07" containerName="payload-processing" Nov 03 22:48:54.275412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275393 4298 cpu_manager.go:414] "containerMap: removing container" podUID="03919c76-4d0f-43a7-b3aa-aa72a98e2e5c" containerName="revoke-keys" Nov 03 22:48:54.275412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275398 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03919c76-4d0f-43a7-b3aa-aa72a98e2e5c" containerName="revoke-keys" Nov 03 22:48:54.275412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275407 4298 cpu_manager.go:414] "containerMap: removing container" podUID="78598387-4616-4d8f-9d4b-acb3f5331af6" containerName="payload-pre-processing" Nov 03 22:48:54.275412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275411 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78598387-4616-4d8f-9d4b-acb3f5331af6" containerName="payload-pre-processing" Nov 03 22:48:54.275637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275427 4298 cpu_manager.go:414] "containerMap: removing container" podUID="df3bfd04-fff1-46d3-865e-b559ac337802" containerName="payload-pre-processing" Nov 03 22:48:54.275637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275432 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df3bfd04-fff1-46d3-865e-b559ac337802" containerName="payload-pre-processing" Nov 03 22:48:54.275637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275524 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="03919c76-4d0f-43a7-b3aa-aa72a98e2e5c" containerName="revoke-keys" Nov 03 22:48:54.275637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275532 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="78598387-4616-4d8f-9d4b-acb3f5331af6" containerName="payload-pre-processing" Nov 03 22:48:54.275637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275539 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="df3bfd04-fff1-46d3-865e-b559ac337802" containerName="payload-pre-processing" Nov 03 22:48:54.275637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275552 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ac7245-3eba-4cfe-bbff-58b56adf7b48" containerName="payload-processing" Nov 03 22:48:54.275637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.275560 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1181ab5-ba26-4ec7-94cb-f9cd9d1eef07" containerName="payload-processing" Nov 03 22:48:54.280057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.280040 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:54.333391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.332551 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:48:54.333391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.332666 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:48:54.337200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.337174 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r"] Nov 03 22:48:54.404926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.404889 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-97965e-tmd7r\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:54.405125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.405065 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-97965e-tmd7r\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:54.405199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.405184 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcf4m\" (UniqueName: \"kubernetes.io/projected/5cd78e5e-2198-4b8f-9f50-97908347681e-kube-api-access-xcf4m\") pod \"maas-api-revoke-keys-e2e-api-a-97965e-tmd7r\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:54.506999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.506961 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-97965e-tmd7r\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:54.507478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.507045 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcf4m\" (UniqueName: \"kubernetes.io/projected/5cd78e5e-2198-4b8f-9f50-97908347681e-kube-api-access-xcf4m\") pod \"maas-api-revoke-keys-e2e-api-a-97965e-tmd7r\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:54.507478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.507132 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-97965e-tmd7r\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:54.507764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.507740 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-97965e-tmd7r\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:54.509515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.509497 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-97965e-tmd7r\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:54.564212 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.564150 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcf4m\" (UniqueName: \"kubernetes.io/projected/5cd78e5e-2198-4b8f-9f50-97908347681e-kube-api-access-xcf4m\") pod \"maas-api-revoke-keys-e2e-api-a-97965e-tmd7r\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:54.594157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.594132 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:54.686197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.686165 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8"] Nov 03 22:48:54.692526 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.692495 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:54.721957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.721746 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8"] Nov 03 22:48:54.811111 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.811078 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:54.811310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.811140 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:54.811456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.811431 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfld6\" (UniqueName: \"kubernetes.io/projected/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-kube-api-access-gfld6\") pod \"maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:54.913033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.912958 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:54.913176 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.913122 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfld6\" (UniqueName: \"kubernetes.io/projected/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-kube-api-access-gfld6\") pod \"maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:54.913249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.913220 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:54.913838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.913818 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:54.915205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.915184 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:54.969106 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:48:54.969071 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd78e5e_2198_4b8f_9f50_97908347681e.slice/crio-aa2df8ba9cdb5b09f82829d179438287b04b258ac40f1754b0f22da34cd49337 WatchSource:0}: Error finding container aa2df8ba9cdb5b09f82829d179438287b04b258ac40f1754b0f22da34cd49337: Status 404 returned error can't find the container with id aa2df8ba9cdb5b09f82829d179438287b04b258ac40f1754b0f22da34cd49337 Nov 03 22:48:54.972485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.972466 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfld6\" (UniqueName: \"kubernetes.io/projected/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-kube-api-access-gfld6\") pod \"maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:54.974354 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:54.974327 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r"] Nov 03 22:48:55.008901 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:55.008856 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:55.379895 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:48:55.379860 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adc28e7_d508_43a8_b9c3_dfac2917b9c9.slice/crio-42dd4d03cbfd3c2f2979a6df9559acf9a4ca2c045f63a908149fc583076f0199 WatchSource:0}: Error finding container 42dd4d03cbfd3c2f2979a6df9559acf9a4ca2c045f63a908149fc583076f0199: Status 404 returned error can't find the container with id 42dd4d03cbfd3c2f2979a6df9559acf9a4ca2c045f63a908149fc583076f0199 Nov 03 22:48:55.384337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:55.384284 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8"] Nov 03 22:48:55.549314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:55.549280 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" event={"ID":"7adc28e7-d508-43a8-b9c3-dfac2917b9c9","Type":"ContainerStarted","Data":"42dd4d03cbfd3c2f2979a6df9559acf9a4ca2c045f63a908149fc583076f0199"} Nov 03 22:48:55.550980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:55.550952 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" event={"ID":"5cd78e5e-2198-4b8f-9f50-97908347681e","Type":"ContainerStarted","Data":"5d4af59fcfe9dbad460d7cadbc0f9694d02d0c41699dfc62a417007712b22310"} Nov 03 22:48:55.551110 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:55.550998 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" event={"ID":"5cd78e5e-2198-4b8f-9f50-97908347681e","Type":"ContainerStarted","Data":"aa2df8ba9cdb5b09f82829d179438287b04b258ac40f1754b0f22da34cd49337"} Nov 03 22:48:55.667236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:55.667138 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" podStartSLOduration=1.667124179 podStartE2EDuration="1.667124179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:55.66422314 +0000 UTC m=+2492.644956793" watchObservedRunningTime="2026-11-03 22:48:55.667124179 +0000 UTC m=+2492.647857832" Nov 03 22:48:56.633937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:56.633838 4298 generic.go:356] "Generic (PLEG): container finished" podID="5cd78e5e-2198-4b8f-9f50-97908347681e" containerID="5d4af59fcfe9dbad460d7cadbc0f9694d02d0c41699dfc62a417007712b22310" exitCode=0 Nov 03 22:48:56.634426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:56.634069 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" event={"ID":"5cd78e5e-2198-4b8f-9f50-97908347681e","Type":"ContainerDied","Data":"5d4af59fcfe9dbad460d7cadbc0f9694d02d0c41699dfc62a417007712b22310"} Nov 03 22:48:56.639743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:56.636446 4298 generic.go:356] "Generic (PLEG): container finished" podID="7adc28e7-d508-43a8-b9c3-dfac2917b9c9" containerID="4f6112509fb7df646aa7697be0faeb5eb42e46cf5884fc27d2ea0ede6ebe6021" exitCode=0 Nov 03 22:48:56.639743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:56.636576 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" event={"ID":"7adc28e7-d508-43a8-b9c3-dfac2917b9c9","Type":"ContainerDied","Data":"4f6112509fb7df646aa7697be0faeb5eb42e46cf5884fc27d2ea0ede6ebe6021"} Nov 03 22:48:57.358485 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.356297 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:57.834428 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.834403 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:57.837999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.837978 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:57.871143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.871117 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-xks-ca\") pod \"5cd78e5e-2198-4b8f-9f50-97908347681e\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " Nov 03 22:48:57.871278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.871233 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-openshift-ca\") pod \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " Nov 03 22:48:57.871278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.871268 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd78e5e-2198-4b8f-9f50-97908347681e-kube-api-access-xcf4m\" (UniqueName: \"kubernetes.io/projected/5cd78e5e-2198-4b8f-9f50-97908347681e-kube-api-access-xcf4m\") pod \"5cd78e5e-2198-4b8f-9f50-97908347681e\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " Nov 03 22:48:57.871399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.871340 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-kube-api-access-gfld6\" (UniqueName: \"kubernetes.io/projected/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-kube-api-access-gfld6\") pod \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " Nov 03 22:48:57.871492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.871477 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-openshift-ca\") pod \"5cd78e5e-2198-4b8f-9f50-97908347681e\" (UID: \"5cd78e5e-2198-4b8f-9f50-97908347681e\") " Nov 03 22:48:57.871568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.871552 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-xks-ca\") pod \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\" (UID: \"7adc28e7-d508-43a8-b9c3-dfac2917b9c9\") " Nov 03 22:48:57.871623 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.871557 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-openshift-ca" pod "7adc28e7-d508-43a8-b9c3-dfac2917b9c9" (UID: "7adc28e7-d508-43a8-b9c3-dfac2917b9c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:57.871885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.871859 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-openshift-ca" pod "5cd78e5e-2198-4b8f-9f50-97908347681e" (UID: "5cd78e5e-2198-4b8f-9f50-97908347681e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:57.872135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.872112 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:57.872232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.872139 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:57.873883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.873625 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-kube-api-access-gfld6" pod "7adc28e7-d508-43a8-b9c3-dfac2917b9c9" (UID: "7adc28e7-d508-43a8-b9c3-dfac2917b9c9"). InnerVolumeSpecName "kube-api-access-gfld6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:57.873883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.873629 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-xks-ca" pod "7adc28e7-d508-43a8-b9c3-dfac2917b9c9" (UID: "7adc28e7-d508-43a8-b9c3-dfac2917b9c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:57.873883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.873836 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-xks-ca" pod "5cd78e5e-2198-4b8f-9f50-97908347681e" (UID: "5cd78e5e-2198-4b8f-9f50-97908347681e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:57.874043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.873922 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd78e5e-2198-4b8f-9f50-97908347681e-kube-api-access-xcf4m" pod "5cd78e5e-2198-4b8f-9f50-97908347681e" (UID: "5cd78e5e-2198-4b8f-9f50-97908347681e"). InnerVolumeSpecName "kube-api-access-xcf4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:57.973027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.972990 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:57.973027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.973025 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd78e5e-2198-4b8f-9f50-97908347681e-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:57.973272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.973036 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcf4m\" (UniqueName: \"kubernetes.io/projected/5cd78e5e-2198-4b8f-9f50-97908347681e-kube-api-access-xcf4m\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:57.973272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:57.973047 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfld6\" (UniqueName: \"kubernetes.io/projected/7adc28e7-d508-43a8-b9c3-dfac2917b9c9-kube-api-access-gfld6\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:58.230746 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.230657 4298 patch_prober.go:29] interesting pod/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:48:58.230940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.230758 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:48:58.652636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.652552 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" Nov 03 22:48:58.652636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.652582 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r" event={"ID":"5cd78e5e-2198-4b8f-9f50-97908347681e","Type":"ContainerDied","Data":"aa2df8ba9cdb5b09f82829d179438287b04b258ac40f1754b0f22da34cd49337"} Nov 03 22:48:58.652636 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.652628 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa2df8ba9cdb5b09f82829d179438287b04b258ac40f1754b0f22da34cd49337" Nov 03 22:48:58.653998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.653977 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" Nov 03 22:48:58.654141 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.654019 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8" event={"ID":"7adc28e7-d508-43a8-b9c3-dfac2917b9c9","Type":"ContainerDied","Data":"42dd4d03cbfd3c2f2979a6df9559acf9a4ca2c045f63a908149fc583076f0199"} Nov 03 22:48:58.654141 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.654058 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42dd4d03cbfd3c2f2979a6df9559acf9a4ca2c045f63a908149fc583076f0199" Nov 03 22:48:58.945039 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.945015 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:48:58.988500 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.988464 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-envoy\") pod \"a85d19cf-d09f-427c-a04c-f97d95723d99\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " Nov 03 22:48:58.988678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.988510 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-token\" (UniqueName: \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-token\") pod \"a85d19cf-d09f-427c-a04c-f97d95723d99\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " Nov 03 22:48:58.988678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.988539 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-data\") pod \"a85d19cf-d09f-427c-a04c-f97d95723d99\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " Nov 03 22:48:58.988678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.988639 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-credential-socket\") pod \"a85d19cf-d09f-427c-a04c-f97d95723d99\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " Nov 03 22:48:58.988927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.988687 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a85d19cf-d09f-427c-a04c-f97d95723d99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85d19cf-d09f-427c-a04c-f97d95723d99-istiod-ca-cert\") pod \"a85d19cf-d09f-427c-a04c-f97d95723d99\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " Nov 03 22:48:58.988927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.988789 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-kube-api-access-xv26n\" (UniqueName: \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-kube-api-access-xv26n\") pod \"a85d19cf-d09f-427c-a04c-f97d95723d99\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " Nov 03 22:48:58.988927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.988878 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-certs\") pod \"a85d19cf-d09f-427c-a04c-f97d95723d99\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " Nov 03 22:48:58.988927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.988904 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-data" pod "a85d19cf-d09f-427c-a04c-f97d95723d99" (UID: "a85d19cf-d09f-427c-a04c-f97d95723d99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:58.989148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.988942 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-credential-socket" pod "a85d19cf-d09f-427c-a04c-f97d95723d99" (UID: "a85d19cf-d09f-427c-a04c-f97d95723d99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:58.989148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.988965 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-podinfo\") pod \"a85d19cf-d09f-427c-a04c-f97d95723d99\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " Nov 03 22:48:58.989148 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.989002 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-socket\") pod \"a85d19cf-d09f-427c-a04c-f97d95723d99\" (UID: \"a85d19cf-d09f-427c-a04c-f97d95723d99\") " Nov 03 22:48:58.989297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.989134 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85d19cf-d09f-427c-a04c-f97d95723d99-istiod-ca-cert" pod "a85d19cf-d09f-427c-a04c-f97d95723d99" (UID: "a85d19cf-d09f-427c-a04c-f97d95723d99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:58.989297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.989224 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-certs" pod "a85d19cf-d09f-427c-a04c-f97d95723d99" (UID: "a85d19cf-d09f-427c-a04c-f97d95723d99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:58.989441 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.989410 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-socket" pod "a85d19cf-d09f-427c-a04c-f97d95723d99" (UID: "a85d19cf-d09f-427c-a04c-f97d95723d99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:58.989586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.989562 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:58.989662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.989592 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:58.989662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.989608 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:58.989662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.989622 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:58.989782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.989637 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85d19cf-d09f-427c-a04c-f97d95723d99-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:58.990839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.990820 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-token" pod "a85d19cf-d09f-427c-a04c-f97d95723d99" (UID: "a85d19cf-d09f-427c-a04c-f97d95723d99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:58.990947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.990928 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-kube-api-access-xv26n" pod "a85d19cf-d09f-427c-a04c-f97d95723d99" (UID: "a85d19cf-d09f-427c-a04c-f97d95723d99"). InnerVolumeSpecName "kube-api-access-xv26n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:58.991122 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.991098 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-envoy" pod "a85d19cf-d09f-427c-a04c-f97d95723d99" (UID: "a85d19cf-d09f-427c-a04c-f97d95723d99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:58.991198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:58.991121 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-podinfo" pod "a85d19cf-d09f-427c-a04c-f97d95723d99" (UID: "a85d19cf-d09f-427c-a04c-f97d95723d99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:48:59.090942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.090901 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv26n\" (UniqueName: \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-kube-api-access-xv26n\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:59.090942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.090935 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:59.090942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.090949 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:59.091178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.090961 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85d19cf-d09f-427c-a04c-f97d95723d99-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:59.662784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.662749 4298 generic.go:356] "Generic (PLEG): container finished" podID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerID="8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594" exitCode=0 Nov 03 22:48:59.662968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.662815 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" Nov 03 22:48:59.662968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.662833 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" event={"ID":"a85d19cf-d09f-427c-a04c-f97d95723d99","Type":"ContainerDied","Data":"8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594"} Nov 03 22:48:59.662968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.662913 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h" event={"ID":"a85d19cf-d09f-427c-a04c-f97d95723d99","Type":"ContainerDied","Data":"bb8725ad925552762b8c207ef54ce8c352f73622fb8e91b557bbce01f1e017e8"} Nov 03 22:48:59.662968 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.662937 4298 scope.go:122] "RemoveContainer" containerID="8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594" Nov 03 22:48:59.673271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.673254 4298 scope.go:122] "RemoveContainer" containerID="9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1" Nov 03 22:48:59.682402 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.682387 4298 scope.go:122] "RemoveContainer" containerID="8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594" Nov 03 22:48:59.682665 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:48:59.682633 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594\": container with ID starting with 8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594 not found: ID does not exist" containerID="8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594" Nov 03 22:48:59.682751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.682673 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594"} err="failed to get container status \"8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594\": rpc error: code = NotFound desc = could not find container \"8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594\": container with ID starting with 8d6cbb36cbd9070fd676ff100f2fb71639699263f6b4459da10ccfc1b77f1594 not found: ID does not exist" Nov 03 22:48:59.682751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.682691 4298 scope.go:122] "RemoveContainer" containerID="9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1" Nov 03 22:48:59.682919 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:48:59.682905 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1\": container with ID starting with 9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1 not found: ID does not exist" containerID="9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1" Nov 03 22:48:59.682956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.682921 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1"} err="failed to get container status \"9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1\": rpc error: code = NotFound desc = could not find container \"9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1\": container with ID starting with 9e4dde1bcd7800431dafba7d8268591856e248c9b0be765964efd98cb1323dc1 not found: ID does not exist" Nov 03 22:48:59.833309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.833276 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h"] Nov 03 22:48:59.838090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:48:59.838065 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f881e3-openshift-default-85cbfb6bdf-6wq7h"] Nov 03 22:49:01.135611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:01.135578 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" path="/var/lib/kubelet/pods/a85d19cf-d09f-427c-a04c-f97d95723d99/volumes" Nov 03 22:49:02.879400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:02.879342 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:05.788786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:05.788750 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs"] Nov 03 22:49:05.899959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:05.899921 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq"] Nov 03 22:49:06.188918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:06.188820 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:06.373776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:06.373746 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:08.010942 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:08.010900 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:08.313576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:08.313498 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8"] Nov 03 22:49:08.427273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:08.427244 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn"] Nov 03 22:49:08.725330 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:08.725295 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:08.910964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:08.910930 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:10.789268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:10.789235 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" podUID="2b394ca3-e524-45da-a176-5e3b1d4d4cf6" containerName="payload-processing" containerID="cri-o://1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b" gracePeriod=25 Nov 03 22:49:10.901035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:10.901000 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" podUID="9e83e8da-abd2-4045-954c-d4e80b249aa8" containerName="payload-pre-processing" containerID="cri-o://b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47" gracePeriod=25 Nov 03 22:49:11.072852 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.072827 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:49:11.175742 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.175717 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:49:11.177004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.176986 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-plugins-config-volume\") pod \"2b394ca3-e524-45da-a176-5e3b1d4d4cf6\" (UID: \"2b394ca3-e524-45da-a176-5e3b1d4d4cf6\") " Nov 03 22:49:11.177083 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.177047 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-kube-api-access-bpdxt\" (UniqueName: \"kubernetes.io/projected/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-kube-api-access-bpdxt\") pod \"2b394ca3-e524-45da-a176-5e3b1d4d4cf6\" (UID: \"2b394ca3-e524-45da-a176-5e3b1d4d4cf6\") " Nov 03 22:49:11.177326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.177307 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-plugins-config-volume" pod "2b394ca3-e524-45da-a176-5e3b1d4d4cf6" (UID: "2b394ca3-e524-45da-a176-5e3b1d4d4cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:11.177487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.177472 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:11.179135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.179115 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-kube-api-access-bpdxt" pod "2b394ca3-e524-45da-a176-5e3b1d4d4cf6" (UID: "2b394ca3-e524-45da-a176-5e3b1d4d4cf6"). InnerVolumeSpecName "kube-api-access-bpdxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:11.278877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.278845 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e83e8da-abd2-4045-954c-d4e80b249aa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e83e8da-abd2-4045-954c-d4e80b249aa8-plugins-config-volume\") pod \"9e83e8da-abd2-4045-954c-d4e80b249aa8\" (UID: \"9e83e8da-abd2-4045-954c-d4e80b249aa8\") " Nov 03 22:49:11.279055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.278921 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e83e8da-abd2-4045-954c-d4e80b249aa8-kube-api-access-nm5kz\" (UniqueName: \"kubernetes.io/projected/9e83e8da-abd2-4045-954c-d4e80b249aa8-kube-api-access-nm5kz\") pod \"9e83e8da-abd2-4045-954c-d4e80b249aa8\" (UID: \"9e83e8da-abd2-4045-954c-d4e80b249aa8\") " Nov 03 22:49:11.279221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.279200 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e83e8da-abd2-4045-954c-d4e80b249aa8-plugins-config-volume" pod "9e83e8da-abd2-4045-954c-d4e80b249aa8" (UID: "9e83e8da-abd2-4045-954c-d4e80b249aa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:11.279532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.279506 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e83e8da-abd2-4045-954c-d4e80b249aa8-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:11.279532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.279527 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpdxt\" (UniqueName: \"kubernetes.io/projected/2b394ca3-e524-45da-a176-5e3b1d4d4cf6-kube-api-access-bpdxt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:11.281064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.281042 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e83e8da-abd2-4045-954c-d4e80b249aa8-kube-api-access-nm5kz" pod "9e83e8da-abd2-4045-954c-d4e80b249aa8" (UID: "9e83e8da-abd2-4045-954c-d4e80b249aa8"). InnerVolumeSpecName "kube-api-access-nm5kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:11.380811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.380734 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm5kz\" (UniqueName: \"kubernetes.io/projected/9e83e8da-abd2-4045-954c-d4e80b249aa8-kube-api-access-nm5kz\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:11.760561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.760521 4298 generic.go:356] "Generic (PLEG): container finished" podID="2b394ca3-e524-45da-a176-5e3b1d4d4cf6" containerID="1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b" exitCode=0 Nov 03 22:49:11.760561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.760564 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" Nov 03 22:49:11.760874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.760605 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" event={"ID":"2b394ca3-e524-45da-a176-5e3b1d4d4cf6","Type":"ContainerDied","Data":"1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b"} Nov 03 22:49:11.760874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.760676 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs" event={"ID":"2b394ca3-e524-45da-a176-5e3b1d4d4cf6","Type":"ContainerDied","Data":"72f2b7dd17045f24dd1d2c28aa291127b489ec919d7add57e16f4bac9d4616d1"} Nov 03 22:49:11.760874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.760707 4298 scope.go:122] "RemoveContainer" containerID="1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b" Nov 03 22:49:11.762206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.762184 4298 generic.go:356] "Generic (PLEG): container finished" podID="9e83e8da-abd2-4045-954c-d4e80b249aa8" containerID="b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47" exitCode=0 Nov 03 22:49:11.762322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.762240 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" Nov 03 22:49:11.762399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.762270 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" event={"ID":"9e83e8da-abd2-4045-954c-d4e80b249aa8","Type":"ContainerDied","Data":"b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47"} Nov 03 22:49:11.762466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.762429 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq" event={"ID":"9e83e8da-abd2-4045-954c-d4e80b249aa8","Type":"ContainerDied","Data":"34c8c5f5bbabdf1759380c5e386827d4ce2fa1ee72e901d0fd08ce03ab67eb2d"} Nov 03 22:49:11.772582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.772564 4298 scope.go:122] "RemoveContainer" containerID="1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b" Nov 03 22:49:11.772927 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:49:11.772909 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b\": container with ID starting with 1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b not found: ID does not exist" containerID="1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b" Nov 03 22:49:11.773003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.772935 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b"} err="failed to get container status \"1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b\": rpc error: code = NotFound desc = could not find container \"1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b\": container with ID starting with 1092a117708c74493ed58ef0b1262bd4d18bdf53b9de16a5b0cebd0fbe8a9c8b not found: ID does not exist" Nov 03 22:49:11.773003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.772952 4298 scope.go:122] "RemoveContainer" containerID="b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47" Nov 03 22:49:11.784355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.784339 4298 scope.go:122] "RemoveContainer" containerID="b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47" Nov 03 22:49:11.784677 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:49:11.784636 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47\": container with ID starting with b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47 not found: ID does not exist" containerID="b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47" Nov 03 22:49:11.784772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.784687 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47"} err="failed to get container status \"b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47\": rpc error: code = NotFound desc = could not find container \"b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47\": container with ID starting with b610c647582d27e39497ac31d24c1f1766c27f27cce377a7e4e7a0eeb3579d47 not found: ID does not exist" Nov 03 22:49:11.929680 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.929634 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs"] Nov 03 22:49:11.980888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:11.980857 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-01bb3a-865d8b98bd-tjfxs"] Nov 03 22:49:12.043071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:12.042997 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq"] Nov 03 22:49:12.048861 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:12.048829 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-01bb3a-6cc9595b55-8dspq"] Nov 03 22:49:12.792164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:12.792123 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:13.136173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.136096 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b394ca3-e524-45da-a176-5e3b1d4d4cf6" path="/var/lib/kubelet/pods/2b394ca3-e524-45da-a176-5e3b1d4d4cf6/volumes" Nov 03 22:49:13.136586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.136543 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e83e8da-abd2-4045-954c-d4e80b249aa8" path="/var/lib/kubelet/pods/9e83e8da-abd2-4045-954c-d4e80b249aa8/volumes" Nov 03 22:49:13.314275 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.314233 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" podUID="1caea868-e30c-40ca-8b1c-2f82fef32b41" containerName="payload-processing" containerID="cri-o://698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605" gracePeriod=25 Nov 03 22:49:13.428582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.428499 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" podUID="4b6f0f1a-5eae-45e8-be70-d250125cb519" containerName="payload-pre-processing" containerID="cri-o://456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336" gracePeriod=25 Nov 03 22:49:13.585262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.585236 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:49:13.708497 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.708472 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:49:13.713557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.713530 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1caea868-e30c-40ca-8b1c-2f82fef32b41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caea868-e30c-40ca-8b1c-2f82fef32b41-plugins-config-volume\") pod \"1caea868-e30c-40ca-8b1c-2f82fef32b41\" (UID: \"1caea868-e30c-40ca-8b1c-2f82fef32b41\") " Nov 03 22:49:13.713823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.713799 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1caea868-e30c-40ca-8b1c-2f82fef32b41-kube-api-access-px87s\" (UniqueName: \"kubernetes.io/projected/1caea868-e30c-40ca-8b1c-2f82fef32b41-kube-api-access-px87s\") pod \"1caea868-e30c-40ca-8b1c-2f82fef32b41\" (UID: \"1caea868-e30c-40ca-8b1c-2f82fef32b41\") " Nov 03 22:49:13.713961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.713923 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1caea868-e30c-40ca-8b1c-2f82fef32b41-plugins-config-volume" pod "1caea868-e30c-40ca-8b1c-2f82fef32b41" (UID: "1caea868-e30c-40ca-8b1c-2f82fef32b41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:13.714451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.714424 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caea868-e30c-40ca-8b1c-2f82fef32b41-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:13.716520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.716493 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1caea868-e30c-40ca-8b1c-2f82fef32b41-kube-api-access-px87s" pod "1caea868-e30c-40ca-8b1c-2f82fef32b41" (UID: "1caea868-e30c-40ca-8b1c-2f82fef32b41"). InnerVolumeSpecName "kube-api-access-px87s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:13.781871 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.781837 4298 generic.go:356] "Generic (PLEG): container finished" podID="4b6f0f1a-5eae-45e8-be70-d250125cb519" containerID="456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336" exitCode=0 Nov 03 22:49:13.782063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.781887 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" Nov 03 22:49:13.782063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.781922 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" event={"ID":"4b6f0f1a-5eae-45e8-be70-d250125cb519","Type":"ContainerDied","Data":"456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336"} Nov 03 22:49:13.782063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.781976 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn" event={"ID":"4b6f0f1a-5eae-45e8-be70-d250125cb519","Type":"ContainerDied","Data":"6f329cf749f24e2e19be7987eae69c7cb6b0c940688a1942808cb2ba217e8300"} Nov 03 22:49:13.782063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.781997 4298 scope.go:122] "RemoveContainer" containerID="456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336" Nov 03 22:49:13.783383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.783359 4298 generic.go:356] "Generic (PLEG): container finished" podID="1caea868-e30c-40ca-8b1c-2f82fef32b41" containerID="698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605" exitCode=0 Nov 03 22:49:13.783493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.783395 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" Nov 03 22:49:13.783493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.783393 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" event={"ID":"1caea868-e30c-40ca-8b1c-2f82fef32b41","Type":"ContainerDied","Data":"698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605"} Nov 03 22:49:13.783573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.783525 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8" event={"ID":"1caea868-e30c-40ca-8b1c-2f82fef32b41","Type":"ContainerDied","Data":"17707d70541476271399aaddccaf9a34ff1a60fcf7dd7eebfa14fe1bff4a0a3c"} Nov 03 22:49:13.792309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.792294 4298 scope.go:122] "RemoveContainer" containerID="456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336" Nov 03 22:49:13.792540 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:49:13.792523 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336\": container with ID starting with 456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336 not found: ID does not exist" containerID="456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336" Nov 03 22:49:13.792589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.792548 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336"} err="failed to get container status \"456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336\": rpc error: code = NotFound desc = could not find container \"456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336\": container with ID starting with 456732a548afb28ca43f6693c65c3ac4837d058055145ef054887198ebacd336 not found: ID does not exist" Nov 03 22:49:13.792589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.792563 4298 scope.go:122] "RemoveContainer" containerID="698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605" Nov 03 22:49:13.807125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.807106 4298 scope.go:122] "RemoveContainer" containerID="698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605" Nov 03 22:49:13.807438 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:49:13.807417 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605\": container with ID starting with 698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605 not found: ID does not exist" containerID="698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605" Nov 03 22:49:13.807488 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.807451 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605"} err="failed to get container status \"698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605\": rpc error: code = NotFound desc = could not find container \"698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605\": container with ID starting with 698381cb2554ddd746a8074889441e9d6c01d8a51745231c4cbe44b5f962f605 not found: ID does not exist" Nov 03 22:49:13.815604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.815582 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b6f0f1a-5eae-45e8-be70-d250125cb519-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6f0f1a-5eae-45e8-be70-d250125cb519-plugins-config-volume\") pod \"4b6f0f1a-5eae-45e8-be70-d250125cb519\" (UID: \"4b6f0f1a-5eae-45e8-be70-d250125cb519\") " Nov 03 22:49:13.815787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.815773 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6f0f1a-5eae-45e8-be70-d250125cb519-kube-api-access-vwlxz\" (UniqueName: \"kubernetes.io/projected/4b6f0f1a-5eae-45e8-be70-d250125cb519-kube-api-access-vwlxz\") pod \"4b6f0f1a-5eae-45e8-be70-d250125cb519\" (UID: \"4b6f0f1a-5eae-45e8-be70-d250125cb519\") " Nov 03 22:49:13.815949 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.815931 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6f0f1a-5eae-45e8-be70-d250125cb519-plugins-config-volume" pod "4b6f0f1a-5eae-45e8-be70-d250125cb519" (UID: "4b6f0f1a-5eae-45e8-be70-d250125cb519"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:13.816233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.816216 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px87s\" (UniqueName: \"kubernetes.io/projected/1caea868-e30c-40ca-8b1c-2f82fef32b41-kube-api-access-px87s\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:13.816276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.816236 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6f0f1a-5eae-45e8-be70-d250125cb519-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:13.817724 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.817707 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6f0f1a-5eae-45e8-be70-d250125cb519-kube-api-access-vwlxz" pod "4b6f0f1a-5eae-45e8-be70-d250125cb519" (UID: "4b6f0f1a-5eae-45e8-be70-d250125cb519"). InnerVolumeSpecName "kube-api-access-vwlxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:13.917371 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.917340 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwlxz\" (UniqueName: \"kubernetes.io/projected/4b6f0f1a-5eae-45e8-be70-d250125cb519-kube-api-access-vwlxz\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:13.992260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.992224 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8"] Nov 03 22:49:13.996749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:13.996722 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97965e-84b4464fc6-wn2b8"] Nov 03 22:49:14.205466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:14.205433 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn"] Nov 03 22:49:14.209867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:14.209847 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97965e-67c7f74999-h6lxn"] Nov 03 22:49:15.135975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:15.135947 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1caea868-e30c-40ca-8b1c-2f82fef32b41" path="/var/lib/kubelet/pods/1caea868-e30c-40ca-8b1c-2f82fef32b41/volumes" Nov 03 22:49:15.136333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:15.136321 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6f0f1a-5eae-45e8-be70-d250125cb519" path="/var/lib/kubelet/pods/4b6f0f1a-5eae-45e8-be70-d250125cb519/volumes" Nov 03 22:49:17.547902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:17.547860 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:17.553130 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:49:17.553094 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:49:17.737538 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:17.737502 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:22.186843 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:22.186808 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:22.420891 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:22.420847 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:26.945310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:26.945277 4298 scope.go:122] "RemoveContainer" containerID="936e875aded9b368624e81d37d7aaab7300b57e354fa79ffbaf2ea4bb0a78b61" Nov 03 22:49:26.955089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:26.955066 4298 scope.go:122] "RemoveContainer" containerID="be61ee6775ca50352d51ff875a16f120090849d7d339038abe15a0e41232f2bc" Nov 03 22:49:26.964824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:26.964803 4298 scope.go:122] "RemoveContainer" containerID="e0469f3f93fc38be1fd4f2f591959db645dca9704547cd5c5917acb54ebf6108" Nov 03 22:49:27.316539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:27.316452 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:27.614480 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:27.614400 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:32.220761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:32.220723 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:36.048245 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:36.048211 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj"] Nov 03 22:49:36.098238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:36.098204 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789a71e0-4x2gj"] Nov 03 22:49:37.113932 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:37.113879 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:37.144614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:37.144578 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c964482-99ad-42ee-888e-f54a236a47e6" path="/var/lib/kubelet/pods/2c964482-99ad-42ee-888e-f54a236a47e6/volumes" Nov 03 22:49:41.907231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:41.907187 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:46.570236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:46.570188 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:46.799615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:46.799568 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:47.552412 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:49:47.552364 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:49:51.597486 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:51.597453 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:56.514835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:49:56.514798 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:01.366625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:01.365883 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:05.541102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:05.541017 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:07.092978 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:07.092928 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:07.318158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:07.318119 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:07.495328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:07.495284 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:07.569458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:07.569422 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8"] Nov 03 22:50:07.619579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:07.619542 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01bb3a-5lzb8"] Nov 03 22:50:08.036627 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:08.036587 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:09.135444 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:09.135411 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7adc28e7-d508-43a8-b9c3-dfac2917b9c9" path="/var/lib/kubelet/pods/7adc28e7-d508-43a8-b9c3-dfac2917b9c9/volumes" Nov 03 22:50:12.463784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:12.463746 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:13.967534 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:13.967502 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:14.154196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:14.154163 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:14.249652 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:14.249569 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm"] Nov 03 22:50:14.249831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:14.249810 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" containerID="cri-o://15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6" gracePeriod=30 Nov 03 22:50:14.358505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:14.358313 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:14.787276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:14.787243 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:15.232939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.232905 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr"] Nov 03 22:50:15.233506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233492 4298 cpu_manager.go:414] "containerMap: removing container" podUID="5cd78e5e-2198-4b8f-9f50-97908347681e" containerName="revoke-keys" Nov 03 22:50:15.233555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233507 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd78e5e-2198-4b8f-9f50-97908347681e" containerName="revoke-keys" Nov 03 22:50:15.233555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233523 4298 cpu_manager.go:414] "containerMap: removing container" podUID="7adc28e7-d508-43a8-b9c3-dfac2917b9c9" containerName="revoke-keys" Nov 03 22:50:15.233555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233529 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adc28e7-d508-43a8-b9c3-dfac2917b9c9" containerName="revoke-keys" Nov 03 22:50:15.233555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233538 4298 cpu_manager.go:414] "containerMap: removing container" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" Nov 03 22:50:15.233555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233543 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" Nov 03 22:50:15.233555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233549 4298 cpu_manager.go:414] "containerMap: removing container" podUID="4b6f0f1a-5eae-45e8-be70-d250125cb519" containerName="payload-pre-processing" Nov 03 22:50:15.233555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233554 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6f0f1a-5eae-45e8-be70-d250125cb519" containerName="payload-pre-processing" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233568 4298 cpu_manager.go:414] "containerMap: removing container" podUID="2b394ca3-e524-45da-a176-5e3b1d4d4cf6" containerName="payload-processing" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233573 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b394ca3-e524-45da-a176-5e3b1d4d4cf6" containerName="payload-processing" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233590 4298 cpu_manager.go:414] "containerMap: removing container" podUID="9e83e8da-abd2-4045-954c-d4e80b249aa8" containerName="payload-pre-processing" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233596 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e83e8da-abd2-4045-954c-d4e80b249aa8" containerName="payload-pre-processing" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233609 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1caea868-e30c-40ca-8b1c-2f82fef32b41" containerName="payload-processing" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233613 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1caea868-e30c-40ca-8b1c-2f82fef32b41" containerName="payload-processing" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233730 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b394ca3-e524-45da-a176-5e3b1d4d4cf6" containerName="payload-processing" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233744 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233750 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e83e8da-abd2-4045-954c-d4e80b249aa8" containerName="payload-pre-processing" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233757 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1caea868-e30c-40ca-8b1c-2f82fef32b41" containerName="payload-processing" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233767 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd78e5e-2198-4b8f-9f50-97908347681e" containerName="revoke-keys" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233773 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="7adc28e7-d508-43a8-b9c3-dfac2917b9c9" containerName="revoke-keys" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233779 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" Nov 03 22:50:15.233800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.233784 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6f0f1a-5eae-45e8-be70-d250125cb519" containerName="payload-pre-processing" Nov 03 22:50:15.237525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.237509 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:15.295144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.295114 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr"] Nov 03 22:50:15.339836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.339801 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:50:15.340012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.339901 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:50:15.342470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.342449 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zr2\" (UniqueName: \"kubernetes.io/projected/1be6f1a6-26ff-4087-a29b-f4f3481bda76-kube-api-access-26zr2\") pod \"maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:15.342712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.342692 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:15.342802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.342733 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:15.443741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.443711 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:15.443935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.443752 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:15.443935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.443874 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zr2\" (UniqueName: \"kubernetes.io/projected/1be6f1a6-26ff-4087-a29b-f4f3481bda76-kube-api-access-26zr2\") pod \"maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:15.444477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.444448 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:15.446372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.446350 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:15.500901 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.500832 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zr2\" (UniqueName: \"kubernetes.io/projected/1be6f1a6-26ff-4087-a29b-f4f3481bda76-kube-api-access-26zr2\") pod \"maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:15.551262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.551236 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:15.920234 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:50:15.920199 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be6f1a6_26ff_4087_a29b_f4f3481bda76.slice/crio-1d97f716a57c1e53059b942bc6e79ebc3f1e643d0f90dca5a4076eb62cfd90d4 WatchSource:0}: Error finding container 1d97f716a57c1e53059b942bc6e79ebc3f1e643d0f90dca5a4076eb62cfd90d4: Status 404 returned error can't find the container with id 1d97f716a57c1e53059b942bc6e79ebc3f1e643d0f90dca5a4076eb62cfd90d4 Nov 03 22:50:15.924383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:15.924350 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr"] Nov 03 22:50:16.212740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:16.212711 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" event={"ID":"1be6f1a6-26ff-4087-a29b-f4f3481bda76","Type":"ContainerStarted","Data":"1d97f716a57c1e53059b942bc6e79ebc3f1e643d0f90dca5a4076eb62cfd90d4"} Nov 03 22:50:17.220845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:17.220813 4298 generic.go:356] "Generic (PLEG): container finished" podID="1be6f1a6-26ff-4087-a29b-f4f3481bda76" containerID="294f41e457ec4beb90506da4508d94fd06b7b8582b3ac0f4fb38d59c3bea1fd3" exitCode=0 Nov 03 22:50:17.221209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:17.220897 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" event={"ID":"1be6f1a6-26ff-4087-a29b-f4f3481bda76","Type":"ContainerDied","Data":"294f41e457ec4beb90506da4508d94fd06b7b8582b3ac0f4fb38d59c3bea1fd3"} Nov 03 22:50:17.552946 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:50:17.552860 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:50:17.856197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:17.856110 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:18.032835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:18.032798 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:18.383809 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:18.383782 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:18.485041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:18.484949 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-xks-ca\") pod \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " Nov 03 22:50:18.485209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:18.485087 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be6f1a6-26ff-4087-a29b-f4f3481bda76-kube-api-access-26zr2\" (UniqueName: \"kubernetes.io/projected/1be6f1a6-26ff-4087-a29b-f4f3481bda76-kube-api-access-26zr2\") pod \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " Nov 03 22:50:18.485209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:18.485149 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-openshift-ca\") pod \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\" (UID: \"1be6f1a6-26ff-4087-a29b-f4f3481bda76\") " Nov 03 22:50:18.485536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:18.485508 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-openshift-ca" pod "1be6f1a6-26ff-4087-a29b-f4f3481bda76" (UID: "1be6f1a6-26ff-4087-a29b-f4f3481bda76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:18.487123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:18.487099 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-xks-ca" pod "1be6f1a6-26ff-4087-a29b-f4f3481bda76" (UID: "1be6f1a6-26ff-4087-a29b-f4f3481bda76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:18.487193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:18.487142 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be6f1a6-26ff-4087-a29b-f4f3481bda76-kube-api-access-26zr2" pod "1be6f1a6-26ff-4087-a29b-f4f3481bda76" (UID: "1be6f1a6-26ff-4087-a29b-f4f3481bda76"). InnerVolumeSpecName "kube-api-access-26zr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:18.587338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:18.587300 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:18.587338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:18.587332 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be6f1a6-26ff-4087-a29b-f4f3481bda76-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:18.587552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:18.587346 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26zr2\" (UniqueName: \"kubernetes.io/projected/1be6f1a6-26ff-4087-a29b-f4f3481bda76-kube-api-access-26zr2\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:19.235946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.235917 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" Nov 03 22:50:19.235946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.235935 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr" event={"ID":"1be6f1a6-26ff-4087-a29b-f4f3481bda76","Type":"ContainerDied","Data":"1d97f716a57c1e53059b942bc6e79ebc3f1e643d0f90dca5a4076eb62cfd90d4"} Nov 03 22:50:19.236171 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.235984 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d97f716a57c1e53059b942bc6e79ebc3f1e643d0f90dca5a4076eb62cfd90d4" Nov 03 22:50:19.523115 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.523089 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:50:19.599165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599136 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-podinfo\") pod \"74935ed0-dfaa-4277-af6d-76b89f6e8970\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " Nov 03 22:50:19.599362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599176 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-envoy\") pod \"74935ed0-dfaa-4277-af6d-76b89f6e8970\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " Nov 03 22:50:19.599362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599196 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-kube-api-access-pcg6l\" (UniqueName: \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-kube-api-access-pcg6l\") pod \"74935ed0-dfaa-4277-af6d-76b89f6e8970\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " Nov 03 22:50:19.599362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599216 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74935ed0-dfaa-4277-af6d-76b89f6e8970-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74935ed0-dfaa-4277-af6d-76b89f6e8970-istiod-ca-cert\") pod \"74935ed0-dfaa-4277-af6d-76b89f6e8970\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " Nov 03 22:50:19.599551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599375 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-socket\") pod \"74935ed0-dfaa-4277-af6d-76b89f6e8970\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " Nov 03 22:50:19.599551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599411 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-data\") pod \"74935ed0-dfaa-4277-af6d-76b89f6e8970\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " Nov 03 22:50:19.599551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599446 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-certs\") pod \"74935ed0-dfaa-4277-af6d-76b89f6e8970\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " Nov 03 22:50:19.599551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599523 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-token\" (UniqueName: \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-token\") pod \"74935ed0-dfaa-4277-af6d-76b89f6e8970\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " Nov 03 22:50:19.599827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599569 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-credential-socket\") pod \"74935ed0-dfaa-4277-af6d-76b89f6e8970\" (UID: \"74935ed0-dfaa-4277-af6d-76b89f6e8970\") " Nov 03 22:50:19.599827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599708 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-socket" pod "74935ed0-dfaa-4277-af6d-76b89f6e8970" (UID: "74935ed0-dfaa-4277-af6d-76b89f6e8970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:19.599827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599724 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-data" pod "74935ed0-dfaa-4277-af6d-76b89f6e8970" (UID: "74935ed0-dfaa-4277-af6d-76b89f6e8970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:19.599827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599744 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74935ed0-dfaa-4277-af6d-76b89f6e8970-istiod-ca-cert" pod "74935ed0-dfaa-4277-af6d-76b89f6e8970" (UID: "74935ed0-dfaa-4277-af6d-76b89f6e8970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:19.600065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599950 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-certs" pod "74935ed0-dfaa-4277-af6d-76b89f6e8970" (UID: "74935ed0-dfaa-4277-af6d-76b89f6e8970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:19.600065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.599967 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-credential-socket" pod "74935ed0-dfaa-4277-af6d-76b89f6e8970" (UID: "74935ed0-dfaa-4277-af6d-76b89f6e8970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:19.600180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.600145 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:19.600180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.600165 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:19.600316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.600180 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:19.600316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.600195 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:19.600316 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.600204 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74935ed0-dfaa-4277-af6d-76b89f6e8970-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:19.601468 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.601441 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-envoy" pod "74935ed0-dfaa-4277-af6d-76b89f6e8970" (UID: "74935ed0-dfaa-4277-af6d-76b89f6e8970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:19.601706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.601686 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-kube-api-access-pcg6l" pod "74935ed0-dfaa-4277-af6d-76b89f6e8970" (UID: "74935ed0-dfaa-4277-af6d-76b89f6e8970"). InnerVolumeSpecName "kube-api-access-pcg6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:19.601825 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.601777 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-podinfo" pod "74935ed0-dfaa-4277-af6d-76b89f6e8970" (UID: "74935ed0-dfaa-4277-af6d-76b89f6e8970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:19.601897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.601879 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-token" pod "74935ed0-dfaa-4277-af6d-76b89f6e8970" (UID: "74935ed0-dfaa-4277-af6d-76b89f6e8970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:19.701991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.701960 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:19.701991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.701986 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcg6l\" (UniqueName: \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-kube-api-access-pcg6l\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:19.701991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.701996 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:19.702225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.702005 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74935ed0-dfaa-4277-af6d-76b89f6e8970-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:19.715820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.715791 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f"] Nov 03 22:50:19.912692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:19.912658 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:20.244703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.244636 4298 generic.go:356] "Generic (PLEG): container finished" podID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerID="15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6" exitCode=0 Nov 03 22:50:20.244896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.244721 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" Nov 03 22:50:20.244896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.244731 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" event={"ID":"74935ed0-dfaa-4277-af6d-76b89f6e8970","Type":"ContainerDied","Data":"15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6"} Nov 03 22:50:20.244896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.244791 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm" event={"ID":"74935ed0-dfaa-4277-af6d-76b89f6e8970","Type":"ContainerDied","Data":"7eacb88399266cfbded59d152629e4d28e70d4082eea932b2ef9d96a0750dd5a"} Nov 03 22:50:20.244896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.244814 4298 scope.go:122] "RemoveContainer" containerID="15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6" Nov 03 22:50:20.255747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.255727 4298 scope.go:122] "RemoveContainer" containerID="a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c" Nov 03 22:50:20.264965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.264945 4298 scope.go:122] "RemoveContainer" containerID="15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6" Nov 03 22:50:20.265233 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:50:20.265215 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6\": container with ID starting with 15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6 not found: ID does not exist" containerID="15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6" Nov 03 22:50:20.265304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.265241 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6"} err="failed to get container status \"15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6\": rpc error: code = NotFound desc = could not find container \"15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6\": container with ID starting with 15be2896239590f21840d6b7e7849d51c43856d322bcd91e26b17937c4de75f6 not found: ID does not exist" Nov 03 22:50:20.265304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.265258 4298 scope.go:122] "RemoveContainer" containerID="a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c" Nov 03 22:50:20.265490 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:50:20.265474 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c\": container with ID starting with a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c not found: ID does not exist" containerID="a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c" Nov 03 22:50:20.265536 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.265495 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c"} err="failed to get container status \"a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c\": rpc error: code = NotFound desc = could not find container \"a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c\": container with ID starting with a872c289c41e29023b71abc224147cbefa79a591661362cc6981950d7726f38c not found: ID does not exist" Nov 03 22:50:20.413703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.413668 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm"] Nov 03 22:50:20.464073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.464043 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-01bb3a-openshift-default-7bd5f7f75f-bwdbm"] Nov 03 22:50:20.797281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.797242 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf"] Nov 03 22:50:20.797849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.797833 4298 cpu_manager.go:414] "containerMap: removing container" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" Nov 03 22:50:20.797849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.797848 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" Nov 03 22:50:20.797996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.797861 4298 cpu_manager.go:414] "containerMap: removing container" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" Nov 03 22:50:20.797996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.797866 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85d19cf-d09f-427c-a04c-f97d95723d99" containerName="istio-proxy" Nov 03 22:50:20.797996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.797888 4298 cpu_manager.go:414] "containerMap: removing container" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" Nov 03 22:50:20.797996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.797893 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" Nov 03 22:50:20.797996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.797909 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1be6f1a6-26ff-4087-a29b-f4f3481bda76" containerName="revoke-keys" Nov 03 22:50:20.797996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.797915 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be6f1a6-26ff-4087-a29b-f4f3481bda76" containerName="revoke-keys" Nov 03 22:50:20.798368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.798019 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" Nov 03 22:50:20.798368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.798030 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be6f1a6-26ff-4087-a29b-f4f3481bda76" containerName="revoke-keys" Nov 03 22:50:20.798368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.798041 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" containerName="istio-proxy" Nov 03 22:50:20.802931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.802913 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:20.856760 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.856731 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:50:20.857055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.856890 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:50:20.859091 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.859070 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf"] Nov 03 22:50:20.918337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.918301 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkxkl\" (UniqueName: \"kubernetes.io/projected/4962b98b-981e-4950-8b9e-7d147fd40b81-kube-api-access-fkxkl\") pod \"maas-api-revoke-sub-4196cd7a-6pmbf\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:20.918337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.918337 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4196cd7a-6pmbf\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:20.918592 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:20.918520 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4196cd7a-6pmbf\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:21.019547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:21.019520 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkxkl\" (UniqueName: \"kubernetes.io/projected/4962b98b-981e-4950-8b9e-7d147fd40b81-kube-api-access-fkxkl\") pod \"maas-api-revoke-sub-4196cd7a-6pmbf\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:21.019547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:21.019550 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4196cd7a-6pmbf\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:21.019806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:21.019683 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4196cd7a-6pmbf\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:21.020303 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:21.020278 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4196cd7a-6pmbf\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:21.021819 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:21.021797 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4196cd7a-6pmbf\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:21.077339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:21.077269 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkxkl\" (UniqueName: \"kubernetes.io/projected/4962b98b-981e-4950-8b9e-7d147fd40b81-kube-api-access-fkxkl\") pod \"maas-api-revoke-sub-4196cd7a-6pmbf\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:21.119349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:21.119320 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:21.141437 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:21.141403 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74935ed0-dfaa-4277-af6d-76b89f6e8970" path="/var/lib/kubelet/pods/74935ed0-dfaa-4277-af6d-76b89f6e8970/volumes" Nov 03 22:50:21.492327 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:50:21.492296 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4962b98b_981e_4950_8b9e_7d147fd40b81.slice/crio-d8961b0848fbe8214a512d71cde0d9c19666c7d9694694243494a7c2b153ae40 WatchSource:0}: Error finding container d8961b0848fbe8214a512d71cde0d9c19666c7d9694694243494a7c2b153ae40: Status 404 returned error can't find the container with id d8961b0848fbe8214a512d71cde0d9c19666c7d9694694243494a7c2b153ae40 Nov 03 22:50:21.497632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:21.497606 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf"] Nov 03 22:50:22.267135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:22.267097 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" event={"ID":"4962b98b-981e-4950-8b9e-7d147fd40b81","Type":"ContainerStarted","Data":"1a7c1d73089b67a633235ebc980a32a7369f5f757b084894a5637b7661b82541"} Nov 03 22:50:22.267544 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:22.267158 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" event={"ID":"4962b98b-981e-4950-8b9e-7d147fd40b81","Type":"ContainerStarted","Data":"d8961b0848fbe8214a512d71cde0d9c19666c7d9694694243494a7c2b153ae40"} Nov 03 22:50:22.391048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:22.390997 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" podStartSLOduration=2.390983361 podStartE2EDuration="2.390983361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:22.386626534 +0000 UTC m=+2579.367360186" watchObservedRunningTime="2026-11-03 22:50:22.390983361 +0000 UTC m=+2579.371717014" Nov 03 22:50:22.536180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:22.536098 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:22.745610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:22.745570 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:23.275017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:23.274983 4298 generic.go:356] "Generic (PLEG): container finished" podID="4962b98b-981e-4950-8b9e-7d147fd40b81" containerID="1a7c1d73089b67a633235ebc980a32a7369f5f757b084894a5637b7661b82541" exitCode=0 Nov 03 22:50:23.275410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:23.275074 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" event={"ID":"4962b98b-981e-4950-8b9e-7d147fd40b81","Type":"ContainerDied","Data":"1a7c1d73089b67a633235ebc980a32a7369f5f757b084894a5637b7661b82541"} Nov 03 22:50:24.433986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:24.433960 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:24.564934 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:24.564853 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-openshift-ca\") pod \"4962b98b-981e-4950-8b9e-7d147fd40b81\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " Nov 03 22:50:24.565107 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:24.564957 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-xks-ca\") pod \"4962b98b-981e-4950-8b9e-7d147fd40b81\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " Nov 03 22:50:24.565107 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:24.565069 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4962b98b-981e-4950-8b9e-7d147fd40b81-kube-api-access-fkxkl\" (UniqueName: \"kubernetes.io/projected/4962b98b-981e-4950-8b9e-7d147fd40b81-kube-api-access-fkxkl\") pod \"4962b98b-981e-4950-8b9e-7d147fd40b81\" (UID: \"4962b98b-981e-4950-8b9e-7d147fd40b81\") " Nov 03 22:50:24.565263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:24.565240 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-openshift-ca" pod "4962b98b-981e-4950-8b9e-7d147fd40b81" (UID: "4962b98b-981e-4950-8b9e-7d147fd40b81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:24.565743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:24.565718 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:24.567208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:24.567180 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-xks-ca" pod "4962b98b-981e-4950-8b9e-7d147fd40b81" (UID: "4962b98b-981e-4950-8b9e-7d147fd40b81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:24.567338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:24.567216 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4962b98b-981e-4950-8b9e-7d147fd40b81-kube-api-access-fkxkl" pod "4962b98b-981e-4950-8b9e-7d147fd40b81" (UID: "4962b98b-981e-4950-8b9e-7d147fd40b81"). InnerVolumeSpecName "kube-api-access-fkxkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:24.667475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:24.667441 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4962b98b-981e-4950-8b9e-7d147fd40b81-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:24.667475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:24.667470 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkxkl\" (UniqueName: \"kubernetes.io/projected/4962b98b-981e-4950-8b9e-7d147fd40b81-kube-api-access-fkxkl\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:25.290206 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:25.290169 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" event={"ID":"4962b98b-981e-4950-8b9e-7d147fd40b81","Type":"ContainerDied","Data":"d8961b0848fbe8214a512d71cde0d9c19666c7d9694694243494a7c2b153ae40"} Nov 03 22:50:25.290399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:25.290234 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8961b0848fbe8214a512d71cde0d9c19666c7d9694694243494a7c2b153ae40" Nov 03 22:50:25.290399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:25.290182 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf" Nov 03 22:50:26.111013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:26.110981 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2"] Nov 03 22:50:26.417230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:26.417113 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:26.598963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:26.598932 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:27.729775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:27.729742 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:28.039196 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:28.039117 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:31.112175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:31.112132 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" podUID="93671667-ad10-4586-8b80-1bb0f18b3a2f" containerName="payload-pre-processing" containerID="cri-o://175c253b7035b434cd67a76dbd64b37ac54b9b626403670f42a8fcbaa68ba7dd" gracePeriod=25 Nov 03 22:50:31.250751 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:50:31.250719 4298 logging.go:55] [core] [Channel #6536 SubChannel #6537]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.43:50052", ServerName: "10.133.0.43:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.43:50052: connect: connection refused" Nov 03 22:50:31.331222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:31.331188 4298 generic.go:356] "Generic (PLEG): container finished" podID="93671667-ad10-4586-8b80-1bb0f18b3a2f" containerID="175c253b7035b434cd67a76dbd64b37ac54b9b626403670f42a8fcbaa68ba7dd" exitCode=0 Nov 03 22:50:31.331386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:31.331241 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" event={"ID":"93671667-ad10-4586-8b80-1bb0f18b3a2f","Type":"ContainerDied","Data":"175c253b7035b434cd67a76dbd64b37ac54b9b626403670f42a8fcbaa68ba7dd"} Nov 03 22:50:31.381503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:31.381447 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:50:31.564900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:31.564869 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93671667-ad10-4586-8b80-1bb0f18b3a2f-kube-api-access-v987z\" (UniqueName: \"kubernetes.io/projected/93671667-ad10-4586-8b80-1bb0f18b3a2f-kube-api-access-v987z\") pod \"93671667-ad10-4586-8b80-1bb0f18b3a2f\" (UID: \"93671667-ad10-4586-8b80-1bb0f18b3a2f\") " Nov 03 22:50:31.565075 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:31.564909 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93671667-ad10-4586-8b80-1bb0f18b3a2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93671667-ad10-4586-8b80-1bb0f18b3a2f-plugins-config-volume\") pod \"93671667-ad10-4586-8b80-1bb0f18b3a2f\" (UID: \"93671667-ad10-4586-8b80-1bb0f18b3a2f\") " Nov 03 22:50:31.565323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:31.565299 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93671667-ad10-4586-8b80-1bb0f18b3a2f-plugins-config-volume" pod "93671667-ad10-4586-8b80-1bb0f18b3a2f" (UID: "93671667-ad10-4586-8b80-1bb0f18b3a2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:31.566972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:31.566950 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93671667-ad10-4586-8b80-1bb0f18b3a2f-kube-api-access-v987z" pod "93671667-ad10-4586-8b80-1bb0f18b3a2f" (UID: "93671667-ad10-4586-8b80-1bb0f18b3a2f"). InnerVolumeSpecName "kube-api-access-v987z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:31.666787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:31.666703 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v987z\" (UniqueName: \"kubernetes.io/projected/93671667-ad10-4586-8b80-1bb0f18b3a2f-kube-api-access-v987z\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:31.666787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:31.666731 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93671667-ad10-4586-8b80-1bb0f18b3a2f-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:32.250702 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:32.250639 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" podUID="93671667-ad10-4586-8b80-1bb0f18b3a2f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.43:50052\" within 1s: context deadline exceeded" Nov 03 22:50:32.338557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:32.338525 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" Nov 03 22:50:32.338749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:32.338561 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2" event={"ID":"93671667-ad10-4586-8b80-1bb0f18b3a2f","Type":"ContainerDied","Data":"93faf86f5ad50c594682d624ef3280170ff9f62b97719547171f3e502019c8f0"} Nov 03 22:50:32.338749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:32.338620 4298 scope.go:122] "RemoveContainer" containerID="175c253b7035b434cd67a76dbd64b37ac54b9b626403670f42a8fcbaa68ba7dd" Nov 03 22:50:32.507467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:32.507385 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2"] Nov 03 22:50:32.512396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:32.512370 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f61714-b95f45ccf-245h2"] Nov 03 22:50:32.526786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:32.526756 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:32.990939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:32.990881 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:33.138781 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.138746 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93671667-ad10-4586-8b80-1bb0f18b3a2f" path="/var/lib/kubelet/pods/93671667-ad10-4586-8b80-1bb0f18b3a2f/volumes" Nov 03 22:50:33.951122 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.951085 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7"] Nov 03 22:50:33.951752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.951737 4298 cpu_manager.go:414] "containerMap: removing container" podUID="93671667-ad10-4586-8b80-1bb0f18b3a2f" containerName="payload-pre-processing" Nov 03 22:50:33.951820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.951753 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93671667-ad10-4586-8b80-1bb0f18b3a2f" containerName="payload-pre-processing" Nov 03 22:50:33.951820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.951783 4298 cpu_manager.go:414] "containerMap: removing container" podUID="4962b98b-981e-4950-8b9e-7d147fd40b81" containerName="revoke-keys" Nov 03 22:50:33.951820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.951788 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4962b98b-981e-4950-8b9e-7d147fd40b81" containerName="revoke-keys" Nov 03 22:50:33.951917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.951872 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="4962b98b-981e-4950-8b9e-7d147fd40b81" containerName="revoke-keys" Nov 03 22:50:33.951917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.951886 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="93671667-ad10-4586-8b80-1bb0f18b3a2f" containerName="payload-pre-processing" Nov 03 22:50:33.956566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.956548 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:33.991891 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.991865 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e3466bb-t2kj7\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:33.992043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.991906 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e3466bb-t2kj7\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:33.992043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:33.991933 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d6jd\" (UniqueName: \"kubernetes.io/projected/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-kube-api-access-8d6jd\") pod \"maas-api-revoke-sub-7e3466bb-t2kj7\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:34.013382 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.013347 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7"] Nov 03 22:50:34.057491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.057460 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:50:34.057630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.057549 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:50:34.093377 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.093349 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e3466bb-t2kj7\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:34.093506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.093388 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e3466bb-t2kj7\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:34.093506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.093412 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d6jd\" (UniqueName: \"kubernetes.io/projected/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-kube-api-access-8d6jd\") pod \"maas-api-revoke-sub-7e3466bb-t2kj7\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:34.094024 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.093994 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e3466bb-t2kj7\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:34.095839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.095814 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e3466bb-t2kj7\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:34.152188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.152161 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d6jd\" (UniqueName: \"kubernetes.io/projected/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-kube-api-access-8d6jd\") pod \"maas-api-revoke-sub-7e3466bb-t2kj7\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:34.269318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.269243 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:34.635477 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:50:34.635451 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6caa58b_8191_4ac1_b316_1e3ec7b4724f.slice/crio-f9727db1a576cada16e22d10b86f4b4a4c537f8408ada46891e5792e7332816d WatchSource:0}: Error finding container f9727db1a576cada16e22d10b86f4b4a4c537f8408ada46891e5792e7332816d: Status 404 returned error can't find the container with id f9727db1a576cada16e22d10b86f4b4a4c537f8408ada46891e5792e7332816d Nov 03 22:50:34.640767 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.640743 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7"] Nov 03 22:50:34.736451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.736416 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" containerID="cri-o://6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436" gracePeriod=45 Nov 03 22:50:34.821429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.821382 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused" Nov 03 22:50:34.983104 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:34.983076 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:50:35.003985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.003958 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-home\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-home\") pod \"59290205-8118-46f3-9998-d49d4395b97f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " Nov 03 22:50:35.004133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.004088 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59290205-8118-46f3-9998-d49d4395b97f-tls-certs\" (UniqueName: \"kubernetes.io/secret/59290205-8118-46f3-9998-d49d4395b97f-tls-certs\") pod \"59290205-8118-46f3-9998-d49d4395b97f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " Nov 03 22:50:35.004194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.004131 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-tmp-dir\") pod \"59290205-8118-46f3-9998-d49d4395b97f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " Nov 03 22:50:35.004194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.004180 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59290205-8118-46f3-9998-d49d4395b97f-kube-api-access-kr54q\" (UniqueName: \"kubernetes.io/projected/59290205-8118-46f3-9998-d49d4395b97f-kube-api-access-kr54q\") pod \"59290205-8118-46f3-9998-d49d4395b97f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " Nov 03 22:50:35.004321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.004208 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-home" pod "59290205-8118-46f3-9998-d49d4395b97f" (UID: "59290205-8118-46f3-9998-d49d4395b97f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.004387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.004346 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-model-cache\") pod \"59290205-8118-46f3-9998-d49d4395b97f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " Nov 03 22:50:35.004442 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.004393 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-tmp-dir" pod "59290205-8118-46f3-9998-d49d4395b97f" (UID: "59290205-8118-46f3-9998-d49d4395b97f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.004442 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.004398 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-dshm\") pod \"59290205-8118-46f3-9998-d49d4395b97f\" (UID: \"59290205-8118-46f3-9998-d49d4395b97f\") " Nov 03 22:50:35.004607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.004585 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-model-cache" pod "59290205-8118-46f3-9998-d49d4395b97f" (UID: "59290205-8118-46f3-9998-d49d4395b97f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.005070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.005047 4298 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-tmp-dir\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.005141 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.005077 4298 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-model-cache\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.005141 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.005092 4298 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-home\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.006417 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.006399 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59290205-8118-46f3-9998-d49d4395b97f-tls-certs" pod "59290205-8118-46f3-9998-d49d4395b97f" (UID: "59290205-8118-46f3-9998-d49d4395b97f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:35.006477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.006428 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-dshm" pod "59290205-8118-46f3-9998-d49d4395b97f" (UID: "59290205-8118-46f3-9998-d49d4395b97f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.006477 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.006463 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59290205-8118-46f3-9998-d49d4395b97f-kube-api-access-kr54q" pod "59290205-8118-46f3-9998-d49d4395b97f" (UID: "59290205-8118-46f3-9998-d49d4395b97f"). InnerVolumeSpecName "kube-api-access-kr54q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:35.106520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.106386 4298 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59290205-8118-46f3-9998-d49d4395b97f-tls-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.106520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.106412 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr54q\" (UniqueName: \"kubernetes.io/projected/59290205-8118-46f3-9998-d49d4395b97f-kube-api-access-kr54q\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.106520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.106422 4298 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59290205-8118-46f3-9998-d49d4395b97f-dshm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.365610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.365574 4298 generic.go:356] "Generic (PLEG): container finished" podID="59290205-8118-46f3-9998-d49d4395b97f" containerID="6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436" exitCode=0 Nov 03 22:50:35.365841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.365779 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" Nov 03 22:50:35.366479 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.366454 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" event={"ID":"59290205-8118-46f3-9998-d49d4395b97f","Type":"ContainerDied","Data":"6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436"} Nov 03 22:50:35.366681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.366498 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f" event={"ID":"59290205-8118-46f3-9998-d49d4395b97f","Type":"ContainerDied","Data":"6474623e8c5b55dc14a560cbe43825e5f0a75545848a6280b1effa1c1206a87b"} Nov 03 22:50:35.366681 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.366521 4298 scope.go:122] "RemoveContainer" containerID="6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436" Nov 03 22:50:35.370354 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.370333 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" event={"ID":"f6caa58b-8191-4ac1-b316-1e3ec7b4724f","Type":"ContainerStarted","Data":"0dc30e18b2739a8761c32079b9fd5ba2c151fe39908b2374ce851074b49dea0f"} Nov 03 22:50:35.370450 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.370379 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" event={"ID":"f6caa58b-8191-4ac1-b316-1e3ec7b4724f","Type":"ContainerStarted","Data":"f9727db1a576cada16e22d10b86f4b4a4c537f8408ada46891e5792e7332816d"} Nov 03 22:50:35.378113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.378093 4298 scope.go:122] "RemoveContainer" containerID="6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436" Nov 03 22:50:35.378404 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:50:35.378385 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436\": container with ID starting with 6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436 not found: ID does not exist" containerID="6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436" Nov 03 22:50:35.378472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.378411 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436"} err="failed to get container status \"6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436\": rpc error: code = NotFound desc = could not find container \"6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436\": container with ID starting with 6d7d1cbc65e65f6f1b4ee34b654f746d60802cafd4b992c3867590805be33436 not found: ID does not exist" Nov 03 22:50:35.538004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.537808 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" podStartSLOduration=2.5377904620000002 podStartE2EDuration="2.537790462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:35.534397862 +0000 UTC m=+2592.515131516" watchObservedRunningTime="2026-11-03 22:50:35.537790462 +0000 UTC m=+2592.518524117" Nov 03 22:50:35.648329 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.648296 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f"] Nov 03 22:50:35.652488 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:35.652469 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/premium-w0-f61714-kserve-57f4f7ffcc-72l5f"] Nov 03 22:50:36.379504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:36.379472 4298 generic.go:356] "Generic (PLEG): container finished" podID="f6caa58b-8191-4ac1-b316-1e3ec7b4724f" containerID="0dc30e18b2739a8761c32079b9fd5ba2c151fe39908b2374ce851074b49dea0f" exitCode=0 Nov 03 22:50:36.380008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:36.379559 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" event={"ID":"f6caa58b-8191-4ac1-b316-1e3ec7b4724f","Type":"ContainerDied","Data":"0dc30e18b2739a8761c32079b9fd5ba2c151fe39908b2374ce851074b49dea0f"} Nov 03 22:50:36.604383 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:36.604336 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:36.800123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:36.800090 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:37.136476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:37.136401 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59290205-8118-46f3-9998-d49d4395b97f" path="/var/lib/kubelet/pods/59290205-8118-46f3-9998-d49d4395b97f/volumes" Nov 03 22:50:37.526121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:37.526091 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:37.646666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:37.644152 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-xks-ca\") pod \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " Nov 03 22:50:37.646666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:37.644425 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-kube-api-access-8d6jd\" (UniqueName: \"kubernetes.io/projected/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-kube-api-access-8d6jd\") pod \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " Nov 03 22:50:37.646666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:37.644669 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-openshift-ca\") pod \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\" (UID: \"f6caa58b-8191-4ac1-b316-1e3ec7b4724f\") " Nov 03 22:50:37.665952 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:37.665915 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-openshift-ca" pod "f6caa58b-8191-4ac1-b316-1e3ec7b4724f" (UID: "f6caa58b-8191-4ac1-b316-1e3ec7b4724f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:37.693972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:37.693897 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-xks-ca" pod "f6caa58b-8191-4ac1-b316-1e3ec7b4724f" (UID: "f6caa58b-8191-4ac1-b316-1e3ec7b4724f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:37.722384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:37.722344 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-kube-api-access-8d6jd" pod "f6caa58b-8191-4ac1-b316-1e3ec7b4724f" (UID: "f6caa58b-8191-4ac1-b316-1e3ec7b4724f"). InnerVolumeSpecName "kube-api-access-8d6jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:37.746603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:37.746560 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d6jd\" (UniqueName: \"kubernetes.io/projected/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-kube-api-access-8d6jd\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:37.746603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:37.746593 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:37.746603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:37.746605 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6caa58b-8191-4ac1-b316-1e3ec7b4724f-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:38.393989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:38.393960 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" Nov 03 22:50:38.394186 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:38.393994 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7" event={"ID":"f6caa58b-8191-4ac1-b316-1e3ec7b4724f","Type":"ContainerDied","Data":"f9727db1a576cada16e22d10b86f4b4a4c537f8408ada46891e5792e7332816d"} Nov 03 22:50:38.394186 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:38.394045 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9727db1a576cada16e22d10b86f4b4a4c537f8408ada46891e5792e7332816d" Nov 03 22:50:38.396063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:38.396036 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:38.633579 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:38.633546 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:38.817231 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:38.817196 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:38.872492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:38.872461 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2"] Nov 03 22:50:38.876652 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:38.876611 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b2fb1-bt9t2"] Nov 03 22:50:39.136375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:39.136296 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceae44a9-cee1-4188-aac6-9e190b7b3e66" path="/var/lib/kubelet/pods/ceae44a9-cee1-4188-aac6-9e190b7b3e66/volumes" Nov 03 22:50:39.329971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:39.329940 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:43.158954 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:43.158906 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:44.791822 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:44.791781 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:45.025743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:45.025711 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:45.224909 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:45.224863 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:45.262714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:45.262674 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp"] Nov 03 22:50:45.266913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:45.266891 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f168-zmjqp"] Nov 03 22:50:45.319785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:45.319747 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt"] Nov 03 22:50:45.320044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:45.320018 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" containerID="cri-o://35b258bf91f6ea199c1dfe0531e6c8bacd869f177b6ed2979fff1c2fecbbf1e5" gracePeriod=30 Nov 03 22:50:45.411193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:45.411162 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:45.718051 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:45.718021 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:46.191901 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.191829 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr"] Nov 03 22:50:46.192475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.192457 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f6caa58b-8191-4ac1-b316-1e3ec7b4724f" containerName="revoke-keys" Nov 03 22:50:46.192540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.192476 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6caa58b-8191-4ac1-b316-1e3ec7b4724f" containerName="revoke-keys" Nov 03 22:50:46.192540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.192491 4298 cpu_manager.go:414] "containerMap: removing container" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" Nov 03 22:50:46.192540 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.192496 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" Nov 03 22:50:46.192669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.192597 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6caa58b-8191-4ac1-b316-1e3ec7b4724f" containerName="revoke-keys" Nov 03 22:50:46.192669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.192605 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="59290205-8118-46f3-9998-d49d4395b97f" containerName="main" Nov 03 22:50:46.197131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.197116 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:46.209451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.209425 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdj7n\" (UniqueName: \"kubernetes.io/projected/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-kube-api-access-vdj7n\") pod \"maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:46.209568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.209531 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:46.209661 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.209624 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:46.256357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.256324 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr"] Nov 03 22:50:46.300222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.300191 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:50:46.300406 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.300380 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:50:46.311279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.311248 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:46.311434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.311341 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:46.311493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.311446 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdj7n\" (UniqueName: \"kubernetes.io/projected/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-kube-api-access-vdj7n\") pod \"maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:46.312108 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.312081 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:46.313817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.313792 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:46.369121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.369087 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdj7n\" (UniqueName: \"kubernetes.io/projected/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-kube-api-access-vdj7n\") pod \"maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:46.510735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.510706 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:46.897000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:46.896975 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr"] Nov 03 22:50:47.136041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:47.136004 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03919c76-4d0f-43a7-b3aa-aa72a98e2e5c" path="/var/lib/kubelet/pods/03919c76-4d0f-43a7-b3aa-aa72a98e2e5c/volumes" Nov 03 22:50:47.512405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:47.512344 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" event={"ID":"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e","Type":"ContainerStarted","Data":"e66617d2c97d937e5da893d49bc100435d0b359b53c2167e60422d561337631c"} Nov 03 22:50:47.512923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:47.512419 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" event={"ID":"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e","Type":"ContainerStarted","Data":"5472f4c0402e6da3a40ad46a5f902b94ee10df5156b03bcc1894a0ac2fbe93b5"} Nov 03 22:50:47.551500 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:50:47.551466 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:50:47.631321 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:47.631270 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" podStartSLOduration=1.631255625 podStartE2EDuration="1.631255625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:47.624890555 +0000 UTC m=+2604.605624208" watchObservedRunningTime="2026-11-03 22:50:47.631255625 +0000 UTC m=+2604.611989277" Nov 03 22:50:48.520429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:48.520390 4298 generic.go:356] "Generic (PLEG): container finished" podID="c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e" containerID="e66617d2c97d937e5da893d49bc100435d0b359b53c2167e60422d561337631c" exitCode=0 Nov 03 22:50:48.520889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:48.520475 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" event={"ID":"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e","Type":"ContainerDied","Data":"e66617d2c97d937e5da893d49bc100435d0b359b53c2167e60422d561337631c"} Nov 03 22:50:49.682997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:49.682969 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:49.759420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:49.759384 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-kube-api-access-vdj7n\" (UniqueName: \"kubernetes.io/projected/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-kube-api-access-vdj7n\") pod \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " Nov 03 22:50:49.759618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:49.759431 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-openshift-ca\") pod \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " Nov 03 22:50:49.759618 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:49.759464 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-xks-ca\") pod \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\" (UID: \"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e\") " Nov 03 22:50:49.759904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:49.759878 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-openshift-ca" pod "c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e" (UID: "c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:49.760073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:49.760052 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:49.761486 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:49.761453 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-kube-api-access-vdj7n" pod "c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e" (UID: "c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e"). InnerVolumeSpecName "kube-api-access-vdj7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:49.761608 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:49.761554 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-xks-ca" pod "c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e" (UID: "c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:49.860879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:49.860800 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdj7n\" (UniqueName: \"kubernetes.io/projected/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-kube-api-access-vdj7n\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:49.860879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:49.860829 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:50.493290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.493202 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:50.535958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.535930 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" Nov 03 22:50:50.536133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.536053 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr" event={"ID":"c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e","Type":"ContainerDied","Data":"5472f4c0402e6da3a40ad46a5f902b94ee10df5156b03bcc1894a0ac2fbe93b5"} Nov 03 22:50:50.536133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.536109 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5472f4c0402e6da3a40ad46a5f902b94ee10df5156b03bcc1894a0ac2fbe93b5" Nov 03 22:50:50.539125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.539093 4298 generic.go:356] "Generic (PLEG): container finished" podID="a535c385-80f0-49ec-870d-7d83ccce3073" containerID="35b258bf91f6ea199c1dfe0531e6c8bacd869f177b6ed2979fff1c2fecbbf1e5" exitCode=0 Nov 03 22:50:50.539274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.539180 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" event={"ID":"a535c385-80f0-49ec-870d-7d83ccce3073","Type":"ContainerDied","Data":"35b258bf91f6ea199c1dfe0531e6c8bacd869f177b6ed2979fff1c2fecbbf1e5"} Nov 03 22:50:50.539274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.539268 4298 scope.go:122] "RemoveContainer" containerID="989c34827f115cc66afea8a033f30be9c8c816375483c25696dec12d6b8af6d0" Nov 03 22:50:50.596031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.596001 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:50:50.671029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671005 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-istio-token\" (UniqueName: \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-istio-token\") pod \"a535c385-80f0-49ec-870d-7d83ccce3073\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " Nov 03 22:50:50.671159 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671047 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-kube-api-access-m2mtt\" (UniqueName: \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-kube-api-access-m2mtt\") pod \"a535c385-80f0-49ec-870d-7d83ccce3073\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " Nov 03 22:50:50.671159 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671088 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a535c385-80f0-49ec-870d-7d83ccce3073-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a535c385-80f0-49ec-870d-7d83ccce3073-istiod-ca-cert\") pod \"a535c385-80f0-49ec-870d-7d83ccce3073\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " Nov 03 22:50:50.671159 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671110 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-envoy\") pod \"a535c385-80f0-49ec-870d-7d83ccce3073\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " Nov 03 22:50:50.671159 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671137 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-certs\") pod \"a535c385-80f0-49ec-870d-7d83ccce3073\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " Nov 03 22:50:50.671387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671163 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-socket\") pod \"a535c385-80f0-49ec-870d-7d83ccce3073\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " Nov 03 22:50:50.671387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671186 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a535c385-80f0-49ec-870d-7d83ccce3073-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a535c385-80f0-49ec-870d-7d83ccce3073-istio-podinfo\") pod \"a535c385-80f0-49ec-870d-7d83ccce3073\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " Nov 03 22:50:50.671387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671286 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-data\") pod \"a535c385-80f0-49ec-870d-7d83ccce3073\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " Nov 03 22:50:50.671543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671388 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-credential-socket\") pod \"a535c385-80f0-49ec-870d-7d83ccce3073\" (UID: \"a535c385-80f0-49ec-870d-7d83ccce3073\") " Nov 03 22:50:50.671543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671432 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-certs" pod "a535c385-80f0-49ec-870d-7d83ccce3073" (UID: "a535c385-80f0-49ec-870d-7d83ccce3073"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:50.671543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671467 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a535c385-80f0-49ec-870d-7d83ccce3073-istiod-ca-cert" pod "a535c385-80f0-49ec-870d-7d83ccce3073" (UID: "a535c385-80f0-49ec-870d-7d83ccce3073"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:50.671729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671678 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-credential-socket" pod "a535c385-80f0-49ec-870d-7d83ccce3073" (UID: "a535c385-80f0-49ec-870d-7d83ccce3073"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:50.671794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671757 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-data" pod "a535c385-80f0-49ec-870d-7d83ccce3073" (UID: "a535c385-80f0-49ec-870d-7d83ccce3073"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:50.671794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.671772 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-socket" pod "a535c385-80f0-49ec-870d-7d83ccce3073" (UID: "a535c385-80f0-49ec-870d-7d83ccce3073"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:50.672114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.672092 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:50.672234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.672117 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a535c385-80f0-49ec-870d-7d83ccce3073-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:50.672234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.672131 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:50.672234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.672144 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:50.672234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.672156 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:50.673804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.673776 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-envoy" pod "a535c385-80f0-49ec-870d-7d83ccce3073" (UID: "a535c385-80f0-49ec-870d-7d83ccce3073"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:50.673923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.673853 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a535c385-80f0-49ec-870d-7d83ccce3073-istio-podinfo" pod "a535c385-80f0-49ec-870d-7d83ccce3073" (UID: "a535c385-80f0-49ec-870d-7d83ccce3073"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:50.673923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.673890 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-kube-api-access-m2mtt" pod "a535c385-80f0-49ec-870d-7d83ccce3073" (UID: "a535c385-80f0-49ec-870d-7d83ccce3073"). InnerVolumeSpecName "kube-api-access-m2mtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:50.674013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.673919 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-istio-token" pod "a535c385-80f0-49ec-870d-7d83ccce3073" (UID: "a535c385-80f0-49ec-870d-7d83ccce3073"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:50.773324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.773290 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:50.773324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.773321 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2mtt\" (UniqueName: \"kubernetes.io/projected/a535c385-80f0-49ec-870d-7d83ccce3073-kube-api-access-m2mtt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:50.773324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.773332 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a535c385-80f0-49ec-870d-7d83ccce3073-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:50.773800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:50.773340 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a535c385-80f0-49ec-870d-7d83ccce3073-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:51.547472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:51.547442 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" Nov 03 22:50:51.547706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:51.547441 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt" event={"ID":"a535c385-80f0-49ec-870d-7d83ccce3073","Type":"ContainerDied","Data":"adb247037e11456b88026e435b115ee29c00110bac64f5123e2745a309b27091"} Nov 03 22:50:51.547706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:51.547578 4298 scope.go:122] "RemoveContainer" containerID="35b258bf91f6ea199c1dfe0531e6c8bacd869f177b6ed2979fff1c2fecbbf1e5" Nov 03 22:50:51.718420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:51.718390 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt"] Nov 03 22:50:51.722457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:51.722437 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1b2fb1-openshift-default-77ff69fc5c-lp9dt"] Nov 03 22:50:52.292783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:52.292743 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:52.487062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:52.487032 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:52.581005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:52.580929 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l"] Nov 03 22:50:52.581156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:52.581133 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" containerID="cri-o://4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8" gracePeriod=30 Nov 03 22:50:52.689666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:52.684620 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:53.136001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.135968 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" path="/var/lib/kubelet/pods/a535c385-80f0-49ec-870d-7d83ccce3073/volumes" Nov 03 22:50:53.385902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.385869 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f"] Nov 03 22:50:53.386670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.386595 4298 cpu_manager.go:414] "containerMap: removing container" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" Nov 03 22:50:53.386670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.386620 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" Nov 03 22:50:53.386848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.386761 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e" containerName="revoke-keys" Nov 03 22:50:53.386848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.386771 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e" containerName="revoke-keys" Nov 03 22:50:53.386964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.386931 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" Nov 03 22:50:53.386964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.386956 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" Nov 03 22:50:53.387063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.386968 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e" containerName="revoke-keys" Nov 03 22:50:53.391762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.391742 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:53.449789 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.449761 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f"] Nov 03 22:50:53.494086 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.494052 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:50:53.494266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.494092 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:50:53.521948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.521918 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-295682-v827f\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:53.522113 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.521987 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-295682-v827f\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:53.522173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.522153 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2c5q\" (UniqueName: \"kubernetes.io/projected/39e35c5b-1acb-434a-b4d1-51f327a2373c-kube-api-access-s2c5q\") pod \"maas-api-revoke-keys-e2e-api-b-295682-v827f\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:53.623763 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.623722 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-295682-v827f\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:53.623966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.623833 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2c5q\" (UniqueName: \"kubernetes.io/projected/39e35c5b-1acb-434a-b4d1-51f327a2373c-kube-api-access-s2c5q\") pod \"maas-api-revoke-keys-e2e-api-b-295682-v827f\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:53.623966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.623961 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-295682-v827f\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:53.624375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.624348 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-295682-v827f\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:53.626211 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.626196 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-295682-v827f\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:53.682392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.682326 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2c5q\" (UniqueName: \"kubernetes.io/projected/39e35c5b-1acb-434a-b4d1-51f327a2373c-kube-api-access-s2c5q\") pod \"maas-api-revoke-keys-e2e-api-b-295682-v827f\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:53.705071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:53.705045 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:54.074062 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:50:54.074029 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e35c5b_1acb_434a_b4d1_51f327a2373c.slice/crio-a07e52af801f5d17e976085b04f8bd327ec09311a690e15281868ca0d707bd57 WatchSource:0}: Error finding container a07e52af801f5d17e976085b04f8bd327ec09311a690e15281868ca0d707bd57: Status 404 returned error can't find the container with id a07e52af801f5d17e976085b04f8bd327ec09311a690e15281868ca0d707bd57 Nov 03 22:50:54.078738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:54.078697 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f"] Nov 03 22:50:54.572439 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:54.572399 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" event={"ID":"39e35c5b-1acb-434a-b4d1-51f327a2373c","Type":"ContainerStarted","Data":"a666a4b0fa3c7f7bbb3fa5855500a109b359796b0137a097f5526a7084115c0b"} Nov 03 22:50:54.572861 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:54.572487 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" event={"ID":"39e35c5b-1acb-434a-b4d1-51f327a2373c","Type":"ContainerStarted","Data":"a07e52af801f5d17e976085b04f8bd327ec09311a690e15281868ca0d707bd57"} Nov 03 22:50:54.686833 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:54.686726 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" podStartSLOduration=1.686710137 podStartE2EDuration="1.686710137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:54.684695774 +0000 UTC m=+2611.665429429" watchObservedRunningTime="2026-11-03 22:50:54.686710137 +0000 UTC m=+2611.667443790" Nov 03 22:50:55.581150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:55.581115 4298 generic.go:356] "Generic (PLEG): container finished" podID="39e35c5b-1acb-434a-b4d1-51f327a2373c" containerID="a666a4b0fa3c7f7bbb3fa5855500a109b359796b0137a097f5526a7084115c0b" exitCode=0 Nov 03 22:50:55.581568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:55.581202 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" event={"ID":"39e35c5b-1acb-434a-b4d1-51f327a2373c","Type":"ContainerDied","Data":"a666a4b0fa3c7f7bbb3fa5855500a109b359796b0137a097f5526a7084115c0b"} Nov 03 22:50:55.710991 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:55.710955 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:56.746828 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:56.746802 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:56.866671 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:56.866569 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-openshift-ca\") pod \"39e35c5b-1acb-434a-b4d1-51f327a2373c\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " Nov 03 22:50:56.866859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:56.866783 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e35c5b-1acb-434a-b4d1-51f327a2373c-kube-api-access-s2c5q\" (UniqueName: \"kubernetes.io/projected/39e35c5b-1acb-434a-b4d1-51f327a2373c-kube-api-access-s2c5q\") pod \"39e35c5b-1acb-434a-b4d1-51f327a2373c\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " Nov 03 22:50:56.866922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:56.866873 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-xks-ca\") pod \"39e35c5b-1acb-434a-b4d1-51f327a2373c\" (UID: \"39e35c5b-1acb-434a-b4d1-51f327a2373c\") " Nov 03 22:50:56.867310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:56.867282 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-openshift-ca" pod "39e35c5b-1acb-434a-b4d1-51f327a2373c" (UID: "39e35c5b-1acb-434a-b4d1-51f327a2373c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:56.867451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:56.867388 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:56.869472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:56.869448 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-xks-ca" pod "39e35c5b-1acb-434a-b4d1-51f327a2373c" (UID: "39e35c5b-1acb-434a-b4d1-51f327a2373c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:56.869578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:56.869536 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e35c5b-1acb-434a-b4d1-51f327a2373c-kube-api-access-s2c5q" pod "39e35c5b-1acb-434a-b4d1-51f327a2373c" (UID: "39e35c5b-1acb-434a-b4d1-51f327a2373c"). InnerVolumeSpecName "kube-api-access-s2c5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:56.968525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:56.968488 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e35c5b-1acb-434a-b4d1-51f327a2373c-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:56.968525 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:56.968521 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2c5q\" (UniqueName: \"kubernetes.io/projected/39e35c5b-1acb-434a-b4d1-51f327a2373c-kube-api-access-s2c5q\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:57.138493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.138397 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:57.321689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.321638 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:57.597863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.597827 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:57.600876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.600854 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" Nov 03 22:50:57.600996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.600926 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f" event={"ID":"39e35c5b-1acb-434a-b4d1-51f327a2373c","Type":"ContainerDied","Data":"a07e52af801f5d17e976085b04f8bd327ec09311a690e15281868ca0d707bd57"} Nov 03 22:50:57.600996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.600978 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a07e52af801f5d17e976085b04f8bd327ec09311a690e15281868ca0d707bd57" Nov 03 22:50:57.877759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.877733 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:50:57.983582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.983544 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-socket\") pod \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " Nov 03 22:50:57.983808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.983597 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-envoy\") pod \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " Nov 03 22:50:57.983808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.983677 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-data\") pod \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " Nov 03 22:50:57.983808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.983760 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-certs\") pod \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " Nov 03 22:50:57.984010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.983815 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-kube-api-access-l2bm9\" (UniqueName: \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-kube-api-access-l2bm9\") pod \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " Nov 03 22:50:57.984010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.983844 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-socket" pod "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" (UID: "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:57.984126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.984021 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-data" pod "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" (UID: "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:57.984126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.984107 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-certs" pod "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" (UID: "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:57.984240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.984129 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-podinfo\") pod \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " Nov 03 22:50:57.984240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.984224 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-token\" (UniqueName: \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-token\") pod \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " Nov 03 22:50:57.984355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.984292 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-credential-socket\") pod \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " Nov 03 22:50:57.984355 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.984322 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istiod-ca-cert\") pod \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\" (UID: \"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2\") " Nov 03 22:50:57.984732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.984707 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-credential-socket" pod "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" (UID: "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:57.984983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.984960 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:57.985067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.984991 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:57.985067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.985006 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:57.985067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.985020 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:57.985067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.985024 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istiod-ca-cert" pod "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" (UID: "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:57.986124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.986098 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-envoy" pod "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" (UID: "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:57.987026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.986988 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-kube-api-access-l2bm9" pod "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" (UID: "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2"). InnerVolumeSpecName "kube-api-access-l2bm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:57.987139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.987056 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-podinfo" pod "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" (UID: "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:57.987139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:57.987085 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-token" pod "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" (UID: "e6bfb644-8044-42e4-a8a7-f2aa4116b1b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:58.086735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.086698 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:58.086735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.086737 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:58.086963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.086752 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:58.086963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.086767 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:58.086963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.086782 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2bm9\" (UniqueName: \"kubernetes.io/projected/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2-kube-api-access-l2bm9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:58.149740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.149604 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:58.618674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.617253 4298 generic.go:356] "Generic (PLEG): container finished" podID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerID="4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8" exitCode=0 Nov 03 22:50:58.618674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.617295 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" event={"ID":"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2","Type":"ContainerDied","Data":"4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8"} Nov 03 22:50:58.618674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.617333 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" event={"ID":"e6bfb644-8044-42e4-a8a7-f2aa4116b1b2","Type":"ContainerDied","Data":"8abb2b269616f21258b43e0bee94e39acd774c227db33801eb5e44a7dd156a19"} Nov 03 22:50:58.618674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.617357 4298 scope.go:122] "RemoveContainer" containerID="4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8" Nov 03 22:50:58.618674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.617515 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l" Nov 03 22:50:58.636013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.635988 4298 scope.go:122] "RemoveContainer" containerID="87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413" Nov 03 22:50:58.645369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.645347 4298 scope.go:122] "RemoveContainer" containerID="4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8" Nov 03 22:50:58.645667 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:50:58.645630 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8\": container with ID starting with 4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8 not found: ID does not exist" containerID="4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8" Nov 03 22:50:58.645726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.645678 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8"} err="failed to get container status \"4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8\": rpc error: code = NotFound desc = could not find container \"4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8\": container with ID starting with 4a8805cd626f70db365ea87d43452b937a860202ca2bd4ea220e4cb87f6e3fe8 not found: ID does not exist" Nov 03 22:50:58.645726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.645697 4298 scope.go:122] "RemoveContainer" containerID="87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413" Nov 03 22:50:58.645929 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:50:58.645910 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413\": container with ID starting with 87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413 not found: ID does not exist" containerID="87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413" Nov 03 22:50:58.645972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.645938 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413"} err="failed to get container status \"87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413\": rpc error: code = NotFound desc = could not find container \"87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413\": container with ID starting with 87f62afe72f1b637260c29a5dc8cdde48f5ba25fd4d69d960ed86bb84ea16413 not found: ID does not exist" Nov 03 22:50:58.789400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.789368 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l"] Nov 03 22:50:58.840118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:58.840089 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c8f168-openshift-default-786989b665-shc2l"] Nov 03 22:50:59.144072 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:50:59.144039 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" path="/var/lib/kubelet/pods/e6bfb644-8044-42e4-a8a7-f2aa4116b1b2/volumes" Nov 03 22:51:00.878665 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:00.877731 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:01.193596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.192376 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:01.349035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349001 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587"] Nov 03 22:51:01.349532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349520 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" Nov 03 22:51:01.349573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349533 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" Nov 03 22:51:01.349573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349547 4298 cpu_manager.go:414] "containerMap: removing container" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" Nov 03 22:51:01.349573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349553 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a535c385-80f0-49ec-870d-7d83ccce3073" containerName="istio-proxy" Nov 03 22:51:01.349573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349569 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" Nov 03 22:51:01.349743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349574 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" Nov 03 22:51:01.349743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349586 4298 cpu_manager.go:414] "containerMap: removing container" podUID="39e35c5b-1acb-434a-b4d1-51f327a2373c" containerName="revoke-keys" Nov 03 22:51:01.349743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349590 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e35c5b-1acb-434a-b4d1-51f327a2373c" containerName="revoke-keys" Nov 03 22:51:01.349743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349711 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" Nov 03 22:51:01.349743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349720 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e35c5b-1acb-434a-b4d1-51f327a2373c" containerName="revoke-keys" Nov 03 22:51:01.349743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.349732 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6bfb644-8044-42e4-a8a7-f2aa4116b1b2" containerName="istio-proxy" Nov 03 22:51:01.354582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.354561 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:01.411703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.411667 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587"] Nov 03 22:51:01.443166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.443120 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-345da9-2p587\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:01.443386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.443324 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtn2h\" (UniqueName: \"kubernetes.io/projected/a89988bb-6b60-424a-b35e-1e889fc9f9ea-kube-api-access-xtn2h\") pod \"maas-api-revoke-keys-e2e-inf-a-345da9-2p587\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:01.443469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.443448 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-345da9-2p587\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:01.448588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.448561 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:51:01.448794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.448744 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:51:01.544294 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.544257 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-345da9-2p587\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:01.544503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.544375 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtn2h\" (UniqueName: \"kubernetes.io/projected/a89988bb-6b60-424a-b35e-1e889fc9f9ea-kube-api-access-xtn2h\") pod \"maas-api-revoke-keys-e2e-inf-a-345da9-2p587\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:01.544503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.544459 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-345da9-2p587\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:01.545102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.545081 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-345da9-2p587\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:01.546613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.546592 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-345da9-2p587\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:01.601354 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.601318 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtn2h\" (UniqueName: \"kubernetes.io/projected/a89988bb-6b60-424a-b35e-1e889fc9f9ea-kube-api-access-xtn2h\") pod \"maas-api-revoke-keys-e2e-inf-a-345da9-2p587\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:01.668011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:01.667975 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:02.037107 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:51:02.037021 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89988bb_6b60_424a_b35e_1e889fc9f9ea.slice/crio-e0ef927ec15b03c2ee859b5d82f0daf5190aad3aa47ce5c1ac8e6d9694524a76 WatchSource:0}: Error finding container e0ef927ec15b03c2ee859b5d82f0daf5190aad3aa47ce5c1ac8e6d9694524a76: Status 404 returned error can't find the container with id e0ef927ec15b03c2ee859b5d82f0daf5190aad3aa47ce5c1ac8e6d9694524a76 Nov 03 22:51:02.041571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:02.041544 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587"] Nov 03 22:51:02.651930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:02.651835 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" event={"ID":"a89988bb-6b60-424a-b35e-1e889fc9f9ea","Type":"ContainerStarted","Data":"29b06420657ca25883d3ba52463d0565c81f782eada869842d7d65dac8d3bc06"} Nov 03 22:51:02.651930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:02.651900 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" event={"ID":"a89988bb-6b60-424a-b35e-1e889fc9f9ea","Type":"ContainerStarted","Data":"e0ef927ec15b03c2ee859b5d82f0daf5190aad3aa47ce5c1ac8e6d9694524a76"} Nov 03 22:51:02.769493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:02.769441 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" podStartSLOduration=1.769425437 podStartE2EDuration="1.769425437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:02.765944416 +0000 UTC m=+2619.746678068" watchObservedRunningTime="2026-11-03 22:51:02.769425437 +0000 UTC m=+2619.750159089" Nov 03 22:51:03.662629 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:03.662590 4298 generic.go:356] "Generic (PLEG): container finished" podID="a89988bb-6b60-424a-b35e-1e889fc9f9ea" containerID="29b06420657ca25883d3ba52463d0565c81f782eada869842d7d65dac8d3bc06" exitCode=0 Nov 03 22:51:03.663144 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:03.662686 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" event={"ID":"a89988bb-6b60-424a-b35e-1e889fc9f9ea","Type":"ContainerDied","Data":"29b06420657ca25883d3ba52463d0565c81f782eada869842d7d65dac8d3bc06"} Nov 03 22:51:03.697409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:03.697376 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:04.838021 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:04.837993 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:04.988068 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:04.988031 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89988bb-6b60-424a-b35e-1e889fc9f9ea-kube-api-access-xtn2h\" (UniqueName: \"kubernetes.io/projected/a89988bb-6b60-424a-b35e-1e889fc9f9ea-kube-api-access-xtn2h\") pod \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " Nov 03 22:51:04.988279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:04.988114 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-xks-ca\") pod \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " Nov 03 22:51:04.988279 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:04.988163 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-openshift-ca\") pod \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\" (UID: \"a89988bb-6b60-424a-b35e-1e889fc9f9ea\") " Nov 03 22:51:04.988784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:04.988754 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-openshift-ca" pod "a89988bb-6b60-424a-b35e-1e889fc9f9ea" (UID: "a89988bb-6b60-424a-b35e-1e889fc9f9ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:04.990678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:04.990621 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89988bb-6b60-424a-b35e-1e889fc9f9ea-kube-api-access-xtn2h" pod "a89988bb-6b60-424a-b35e-1e889fc9f9ea" (UID: "a89988bb-6b60-424a-b35e-1e889fc9f9ea"). InnerVolumeSpecName "kube-api-access-xtn2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:04.990780 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:04.990687 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-xks-ca" pod "a89988bb-6b60-424a-b35e-1e889fc9f9ea" (UID: "a89988bb-6b60-424a-b35e-1e889fc9f9ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:05.049495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.049462 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7"] Nov 03 22:51:05.053703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.053680 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24"] Nov 03 22:51:05.057984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.057966 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9"] Nov 03 22:51:05.062213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.062188 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2225ce43-7jfr7"] Nov 03 22:51:05.089977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.089947 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtn2h\" (UniqueName: \"kubernetes.io/projected/a89988bb-6b60-424a-b35e-1e889fc9f9ea-kube-api-access-xtn2h\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:05.089977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.089974 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:05.090162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.089986 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a89988bb-6b60-424a-b35e-1e889fc9f9ea-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:05.099700 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.099664 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087bac71-n5s24"] Nov 03 22:51:05.105408 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.105384 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0388afdf-tjdw9"] Nov 03 22:51:05.136124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.136094 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3538021b-3161-486c-b39d-ebccba8905af" path="/var/lib/kubelet/pods/3538021b-3161-486c-b39d-ebccba8905af/volumes" Nov 03 22:51:05.136570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.136549 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6983c091-4d4a-4001-84cb-b89bcef6a2f3" path="/var/lib/kubelet/pods/6983c091-4d4a-4001-84cb-b89bcef6a2f3/volumes" Nov 03 22:51:05.137134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.137113 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f9ff8b-8274-4fcc-abd7-a28523f93055" path="/var/lib/kubelet/pods/d2f9ff8b-8274-4fcc-abd7-a28523f93055/volumes" Nov 03 22:51:05.677764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.677736 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" Nov 03 22:51:05.677961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.677766 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587" event={"ID":"a89988bb-6b60-424a-b35e-1e889fc9f9ea","Type":"ContainerDied","Data":"e0ef927ec15b03c2ee859b5d82f0daf5190aad3aa47ce5c1ac8e6d9694524a76"} Nov 03 22:51:05.677961 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.677828 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0ef927ec15b03c2ee859b5d82f0daf5190aad3aa47ce5c1ac8e6d9694524a76" Nov 03 22:51:05.708201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.708158 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj"] Nov 03 22:51:05.708731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.708716 4298 cpu_manager.go:414] "containerMap: removing container" podUID="a89988bb-6b60-424a-b35e-1e889fc9f9ea" containerName="revoke-keys" Nov 03 22:51:05.708774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.708732 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89988bb-6b60-424a-b35e-1e889fc9f9ea" containerName="revoke-keys" Nov 03 22:51:05.708836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.708827 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89988bb-6b60-424a-b35e-1e889fc9f9ea" containerName="revoke-keys" Nov 03 22:51:05.713551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.713535 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:05.766435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.766389 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:51:05.766635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.766461 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:51:05.769886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.769855 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj"] Nov 03 22:51:05.815926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.815880 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8"] Nov 03 22:51:05.820191 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.820170 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:05.886319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.884040 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8"] Nov 03 22:51:05.900913 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.900873 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3010b426-rgnjj\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:05.901097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.901021 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3010b426-rgnjj\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:05.901258 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:05.901234 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t69kw\" (UniqueName: \"kubernetes.io/projected/27d77a26-67b3-4553-bacd-bc88287d4672-kube-api-access-t69kw\") pod \"maas-api-revoke-sub-3010b426-rgnjj\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:06.002883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.002837 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff4be605-v6mn8\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:06.003073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.002905 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t69kw\" (UniqueName: \"kubernetes.io/projected/27d77a26-67b3-4553-bacd-bc88287d4672-kube-api-access-t69kw\") pod \"maas-api-revoke-sub-3010b426-rgnjj\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:06.003073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.002978 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skhlf\" (UniqueName: \"kubernetes.io/projected/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-kube-api-access-skhlf\") pod \"maas-api-revoke-sub-ff4be605-v6mn8\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:06.003185 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.003169 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3010b426-rgnjj\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:06.003232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.003221 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3010b426-rgnjj\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:06.003320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.003251 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff4be605-v6mn8\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:06.004087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.004056 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3010b426-rgnjj\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:06.004826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.004793 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:06.011259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.011231 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3010b426-rgnjj\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:06.048244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.048208 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x"] Nov 03 22:51:06.052962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.052926 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b25d69-7cl9x"] Nov 03 22:51:06.060788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.060760 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t69kw\" (UniqueName: \"kubernetes.io/projected/27d77a26-67b3-4553-bacd-bc88287d4672-kube-api-access-t69kw\") pod \"maas-api-revoke-sub-3010b426-rgnjj\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:06.105190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.105146 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff4be605-v6mn8\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:06.105405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.105251 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff4be605-v6mn8\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:06.105405 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.105344 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhlf\" (UniqueName: \"kubernetes.io/projected/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-kube-api-access-skhlf\") pod \"maas-api-revoke-sub-ff4be605-v6mn8\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:06.106769 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.106729 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff4be605-v6mn8\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:06.109487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.109426 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff4be605-v6mn8\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:06.162752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.162716 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhlf\" (UniqueName: \"kubernetes.io/projected/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-kube-api-access-skhlf\") pod \"maas-api-revoke-sub-ff4be605-v6mn8\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:06.326890 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.326802 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:06.436667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.434091 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:06.698369 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:51:06.698341 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d77a26_67b3_4553_bacd_bc88287d4672.slice/crio-4463199f2716b3cd15a2befa28a2ce3bb9977499c968947e991032d4d4041e7b WatchSource:0}: Error finding container 4463199f2716b3cd15a2befa28a2ce3bb9977499c968947e991032d4d4041e7b: Status 404 returned error can't find the container with id 4463199f2716b3cd15a2befa28a2ce3bb9977499c968947e991032d4d4041e7b Nov 03 22:51:06.702530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.702507 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj"] Nov 03 22:51:06.859082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:06.856171 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8"] Nov 03 22:51:06.926737 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:51:06.926706 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c1b4fe_4e0c_4b1e_95d1_3c8e96b0b3b4.slice/crio-1b124e56897db4675f0417663ea0f51729f4c8fb1e0ffd6abe41cf8b390a81d8 WatchSource:0}: Error finding container 1b124e56897db4675f0417663ea0f51729f4c8fb1e0ffd6abe41cf8b390a81d8: Status 404 returned error can't find the container with id 1b124e56897db4675f0417663ea0f51729f4c8fb1e0ffd6abe41cf8b390a81d8 Nov 03 22:51:07.138175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:07.138150 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c8f51f-60ce-44a7-8e67-310faeae985b" path="/var/lib/kubelet/pods/e9c8f51f-60ce-44a7-8e67-310faeae985b/volumes" Nov 03 22:51:07.590504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:07.590431 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:07.697350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:07.697315 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" event={"ID":"27d77a26-67b3-4553-bacd-bc88287d4672","Type":"ContainerStarted","Data":"9bc4c4f8ca61f491e5d862bb74ff076f14fd0d4c6b6fdd141c17aa22267bd520"} Nov 03 22:51:07.697514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:07.697375 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" event={"ID":"27d77a26-67b3-4553-bacd-bc88287d4672","Type":"ContainerStarted","Data":"4463199f2716b3cd15a2befa28a2ce3bb9977499c968947e991032d4d4041e7b"} Nov 03 22:51:07.698918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:07.698894 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" event={"ID":"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4","Type":"ContainerStarted","Data":"fb6a5dc1cde91eb12561ef7cc1dc41388946e6b7db6d071c3db07adf649564f6"} Nov 03 22:51:07.699035 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:07.698934 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" event={"ID":"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4","Type":"ContainerStarted","Data":"1b124e56897db4675f0417663ea0f51729f4c8fb1e0ffd6abe41cf8b390a81d8"} Nov 03 22:51:07.856083 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:07.855993 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:07.932194 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:07.932141 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" podStartSLOduration=2.9321287639999998 podStartE2EDuration="2.932128764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:07.926833316 +0000 UTC m=+2624.907566969" watchObservedRunningTime="2026-11-03 22:51:07.932128764 +0000 UTC m=+2624.912862417" Nov 03 22:51:08.158342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:08.158263 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:08.202562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:08.202527 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r"] Nov 03 22:51:08.206800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:08.206771 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-97965e-tmd7r"] Nov 03 22:51:08.706720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:08.706689 4298 generic.go:356] "Generic (PLEG): container finished" podID="27d77a26-67b3-4553-bacd-bc88287d4672" containerID="9bc4c4f8ca61f491e5d862bb74ff076f14fd0d4c6b6fdd141c17aa22267bd520" exitCode=0 Nov 03 22:51:08.706897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:08.706775 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" event={"ID":"27d77a26-67b3-4553-bacd-bc88287d4672","Type":"ContainerDied","Data":"9bc4c4f8ca61f491e5d862bb74ff076f14fd0d4c6b6fdd141c17aa22267bd520"} Nov 03 22:51:08.708461 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:08.708435 4298 generic.go:356] "Generic (PLEG): container finished" podID="01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4" containerID="fb6a5dc1cde91eb12561ef7cc1dc41388946e6b7db6d071c3db07adf649564f6" exitCode=0 Nov 03 22:51:08.708564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:08.708524 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" event={"ID":"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4","Type":"ContainerDied","Data":"fb6a5dc1cde91eb12561ef7cc1dc41388946e6b7db6d071c3db07adf649564f6"} Nov 03 22:51:08.711094 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:08.711076 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:08.870455 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:08.870410 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:09.042555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.042404 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-xks-ca\") pod \"27d77a26-67b3-4553-bacd-bc88287d4672\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " Nov 03 22:51:09.042555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.042440 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-openshift-ca\") pod \"27d77a26-67b3-4553-bacd-bc88287d4672\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " Nov 03 22:51:09.042555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.042541 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d77a26-67b3-4553-bacd-bc88287d4672-kube-api-access-t69kw\" (UniqueName: \"kubernetes.io/projected/27d77a26-67b3-4553-bacd-bc88287d4672-kube-api-access-t69kw\") pod \"27d77a26-67b3-4553-bacd-bc88287d4672\" (UID: \"27d77a26-67b3-4553-bacd-bc88287d4672\") " Nov 03 22:51:09.043519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.043492 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-openshift-ca" pod "27d77a26-67b3-4553-bacd-bc88287d4672" (UID: "27d77a26-67b3-4553-bacd-bc88287d4672"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:09.043725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.043707 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:09.045689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.045636 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d77a26-67b3-4553-bacd-bc88287d4672-kube-api-access-t69kw" pod "27d77a26-67b3-4553-bacd-bc88287d4672" (UID: "27d77a26-67b3-4553-bacd-bc88287d4672"). InnerVolumeSpecName "kube-api-access-t69kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:09.045790 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.045743 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-xks-ca" pod "27d77a26-67b3-4553-bacd-bc88287d4672" (UID: "27d77a26-67b3-4553-bacd-bc88287d4672"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:09.162667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.157157 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d77a26-67b3-4553-bacd-bc88287d4672-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:09.162667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.157203 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t69kw\" (UniqueName: \"kubernetes.io/projected/27d77a26-67b3-4553-bacd-bc88287d4672-kube-api-access-t69kw\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:09.174712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.174654 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd78e5e-2198-4b8f-9f50-97908347681e" path="/var/lib/kubelet/pods/5cd78e5e-2198-4b8f-9f50-97908347681e/volumes" Nov 03 22:51:09.716159 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.716129 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" Nov 03 22:51:09.716352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.716170 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj" event={"ID":"27d77a26-67b3-4553-bacd-bc88287d4672","Type":"ContainerDied","Data":"4463199f2716b3cd15a2befa28a2ce3bb9977499c968947e991032d4d4041e7b"} Nov 03 22:51:09.716352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.716239 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4463199f2716b3cd15a2befa28a2ce3bb9977499c968947e991032d4d4041e7b" Nov 03 22:51:09.872267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.872241 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:09.974214 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.974134 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-xks-ca\") pod \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " Nov 03 22:51:09.974387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.974255 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-kube-api-access-skhlf\" (UniqueName: \"kubernetes.io/projected/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-kube-api-access-skhlf\") pod \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " Nov 03 22:51:09.974387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.974359 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-openshift-ca\") pod \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\" (UID: \"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4\") " Nov 03 22:51:09.974788 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.974762 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-openshift-ca" pod "01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4" (UID: "01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:09.974946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.974873 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:09.976552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.976527 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-xks-ca" pod "01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4" (UID: "01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:09.976552 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:09.976541 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-kube-api-access-skhlf" pod "01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4" (UID: "01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4"). InnerVolumeSpecName "kube-api-access-skhlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:10.076238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:10.076195 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:10.076238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:10.076234 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skhlf\" (UniqueName: \"kubernetes.io/projected/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4-kube-api-access-skhlf\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:10.723734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:10.723703 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" Nov 03 22:51:10.723940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:10.723734 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8" event={"ID":"01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4","Type":"ContainerDied","Data":"1b124e56897db4675f0417663ea0f51729f4c8fb1e0ffd6abe41cf8b390a81d8"} Nov 03 22:51:10.723940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:10.723781 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b124e56897db4675f0417663ea0f51729f4c8fb1e0ffd6abe41cf8b390a81d8" Nov 03 22:51:13.185669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:13.183632 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:13.366698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:13.366636 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:13.563150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:13.563099 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:14.835897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:14.835863 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:15.029189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.029155 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:15.148741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.148631 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb"] Nov 03 22:51:15.148970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.148921 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" containerID="cri-o://37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979" gracePeriod=30 Nov 03 22:51:15.229666 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.225671 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:15.271135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.270908 4298 patch_prober.go:29] interesting pod/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:51:15.271135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.270987 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:51:15.405094 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.404998 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:15.977225 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.977189 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp"] Nov 03 22:51:15.977739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.977724 4298 cpu_manager.go:414] "containerMap: removing container" podUID="27d77a26-67b3-4553-bacd-bc88287d4672" containerName="revoke-keys" Nov 03 22:51:15.977800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.977739 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d77a26-67b3-4553-bacd-bc88287d4672" containerName="revoke-keys" Nov 03 22:51:15.977800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.977784 4298 cpu_manager.go:414] "containerMap: removing container" podUID="01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4" containerName="revoke-keys" Nov 03 22:51:15.977800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.977792 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4" containerName="revoke-keys" Nov 03 22:51:15.977906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.977887 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4" containerName="revoke-keys" Nov 03 22:51:15.977906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.977899 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d77a26-67b3-4553-bacd-bc88287d4672" containerName="revoke-keys" Nov 03 22:51:15.981491 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:15.981476 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:16.039656 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.039621 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp"] Nov 03 22:51:16.084422 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.084393 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:51:16.084591 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.084479 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:51:16.111740 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.111706 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-472b46-58kxp\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:16.111952 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.111779 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-472b46-58kxp\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:16.112023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.111979 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8v72\" (UniqueName: \"kubernetes.io/projected/155be8a8-7a3f-4f38-898d-1353747ca4c8-kube-api-access-g8v72\") pod \"maas-api-revoke-keys-e2e-api-b-472b46-58kxp\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:16.213180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.213147 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8v72\" (UniqueName: \"kubernetes.io/projected/155be8a8-7a3f-4f38-898d-1353747ca4c8-kube-api-access-g8v72\") pod \"maas-api-revoke-keys-e2e-api-b-472b46-58kxp\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:16.213490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.213467 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-472b46-58kxp\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:16.213589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.213573 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-472b46-58kxp\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:16.214080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.214055 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-472b46-58kxp\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:16.215831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.215808 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-472b46-58kxp\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:16.270560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.270483 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8v72\" (UniqueName: \"kubernetes.io/projected/155be8a8-7a3f-4f38-898d-1353747ca4c8-kube-api-access-g8v72\") pod \"maas-api-revoke-keys-e2e-api-b-472b46-58kxp\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:16.295267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.295233 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:16.662095 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:51:16.662064 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155be8a8_7a3f_4f38_898d_1353747ca4c8.slice/crio-d10406a41d48fadfd1d5c6a95cba6a199fe751b4a8f37eba97d5ce28b5f69587 WatchSource:0}: Error finding container d10406a41d48fadfd1d5c6a95cba6a199fe751b4a8f37eba97d5ce28b5f69587: Status 404 returned error can't find the container with id d10406a41d48fadfd1d5c6a95cba6a199fe751b4a8f37eba97d5ce28b5f69587 Nov 03 22:51:16.665719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.665693 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp"] Nov 03 22:51:16.679048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.679021 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv"] Nov 03 22:51:16.679242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.679221 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" containerID="cri-o://24318ed91e9a2ac3b4f2fa5d1d2efd3736deeb49f338e55bd65ab31c4be98052" gracePeriod=30 Nov 03 22:51:16.766729 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:16.766698 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" event={"ID":"155be8a8-7a3f-4f38-898d-1353747ca4c8","Type":"ContainerStarted","Data":"d10406a41d48fadfd1d5c6a95cba6a199fe751b4a8f37eba97d5ce28b5f69587"} Nov 03 22:51:17.356734 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:17.356693 4298 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:51:17.357218 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:17.356762 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:51:17.565444 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:51:17.565404 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:51:17.625467 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:17.625361 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:17.776827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:17.776777 4298 generic.go:356] "Generic (PLEG): container finished" podID="155be8a8-7a3f-4f38-898d-1353747ca4c8" containerID="13274c397259e47b01124e560c8bb0e73334df7d7d879a9941362692311425dd" exitCode=0 Nov 03 22:51:17.777046 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:17.776925 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" event={"ID":"155be8a8-7a3f-4f38-898d-1353747ca4c8","Type":"ContainerDied","Data":"13274c397259e47b01124e560c8bb0e73334df7d7d879a9941362692311425dd"} Nov 03 22:51:18.278220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:18.278180 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:18.932076 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:18.932056 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:18.988851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:18.988819 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-openshift-ca\") pod \"155be8a8-7a3f-4f38-898d-1353747ca4c8\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " Nov 03 22:51:18.988997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:18.988918 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155be8a8-7a3f-4f38-898d-1353747ca4c8-kube-api-access-g8v72\" (UniqueName: \"kubernetes.io/projected/155be8a8-7a3f-4f38-898d-1353747ca4c8-kube-api-access-g8v72\") pod \"155be8a8-7a3f-4f38-898d-1353747ca4c8\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " Nov 03 22:51:18.989058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:18.989004 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-xks-ca\") pod \"155be8a8-7a3f-4f38-898d-1353747ca4c8\" (UID: \"155be8a8-7a3f-4f38-898d-1353747ca4c8\") " Nov 03 22:51:18.989209 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:18.989189 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-openshift-ca" pod "155be8a8-7a3f-4f38-898d-1353747ca4c8" (UID: "155be8a8-7a3f-4f38-898d-1353747ca4c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:18.989387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:18.989360 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:18.991071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:18.991052 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155be8a8-7a3f-4f38-898d-1353747ca4c8-kube-api-access-g8v72" pod "155be8a8-7a3f-4f38-898d-1353747ca4c8" (UID: "155be8a8-7a3f-4f38-898d-1353747ca4c8"). InnerVolumeSpecName "kube-api-access-g8v72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:18.991171 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:18.991119 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-xks-ca" pod "155be8a8-7a3f-4f38-898d-1353747ca4c8" (UID: "155be8a8-7a3f-4f38-898d-1353747ca4c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:19.090469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:19.090389 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155be8a8-7a3f-4f38-898d-1353747ca4c8-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:19.090469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:19.090428 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8v72\" (UniqueName: \"kubernetes.io/projected/155be8a8-7a3f-4f38-898d-1353747ca4c8-kube-api-access-g8v72\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:19.298338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:19.298308 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:19.577331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:19.577292 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:19.756494 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:19.756456 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:19.791259 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:19.791228 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" Nov 03 22:51:19.791421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:19.791261 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp" event={"ID":"155be8a8-7a3f-4f38-898d-1353747ca4c8","Type":"ContainerDied","Data":"d10406a41d48fadfd1d5c6a95cba6a199fe751b4a8f37eba97d5ce28b5f69587"} Nov 03 22:51:19.791421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:19.791309 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d10406a41d48fadfd1d5c6a95cba6a199fe751b4a8f37eba97d5ce28b5f69587" Nov 03 22:51:19.817866 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:19.817836 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587"] Nov 03 22:51:19.867889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:19.867823 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-345da9-2p587"] Nov 03 22:51:20.278254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.278219 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:20.421990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.421921 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:51:20.507360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.507331 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-kube-api-access-vsbpd\" (UniqueName: \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-kube-api-access-vsbpd\") pod \"e6c3301f-7e55-4e81-b5ce-420873ef1242\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " Nov 03 22:51:20.507584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.507368 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-certs\") pod \"e6c3301f-7e55-4e81-b5ce-420873ef1242\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " Nov 03 22:51:20.507584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.507425 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-socket\") pod \"e6c3301f-7e55-4e81-b5ce-420873ef1242\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " Nov 03 22:51:20.507584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.507487 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-podinfo\") pod \"e6c3301f-7e55-4e81-b5ce-420873ef1242\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " Nov 03 22:51:20.507807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.507590 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c3301f-7e55-4e81-b5ce-420873ef1242-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c3301f-7e55-4e81-b5ce-420873ef1242-istiod-ca-cert\") pod \"e6c3301f-7e55-4e81-b5ce-420873ef1242\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " Nov 03 22:51:20.507807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.507637 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-token\" (UniqueName: \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-token\") pod \"e6c3301f-7e55-4e81-b5ce-420873ef1242\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " Nov 03 22:51:20.507807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.507703 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-data\") pod \"e6c3301f-7e55-4e81-b5ce-420873ef1242\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " Nov 03 22:51:20.507807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.507713 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-certs" pod "e6c3301f-7e55-4e81-b5ce-420873ef1242" (UID: "e6c3301f-7e55-4e81-b5ce-420873ef1242"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:20.507807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.507738 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-envoy\") pod \"e6c3301f-7e55-4e81-b5ce-420873ef1242\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " Nov 03 22:51:20.508084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.507813 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-credential-socket\") pod \"e6c3301f-7e55-4e81-b5ce-420873ef1242\" (UID: \"e6c3301f-7e55-4e81-b5ce-420873ef1242\") " Nov 03 22:51:20.508084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.507823 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-socket" pod "e6c3301f-7e55-4e81-b5ce-420873ef1242" (UID: "e6c3301f-7e55-4e81-b5ce-420873ef1242"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:20.508197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.508157 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-credential-socket" pod "e6c3301f-7e55-4e81-b5ce-420873ef1242" (UID: "e6c3301f-7e55-4e81-b5ce-420873ef1242"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:20.508277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.508250 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c3301f-7e55-4e81-b5ce-420873ef1242-istiod-ca-cert" pod "e6c3301f-7e55-4e81-b5ce-420873ef1242" (UID: "e6c3301f-7e55-4e81-b5ce-420873ef1242"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:20.508341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.508314 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-data" pod "e6c3301f-7e55-4e81-b5ce-420873ef1242" (UID: "e6c3301f-7e55-4e81-b5ce-420873ef1242"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:20.508513 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.508493 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:20.508593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.508520 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:20.508593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.508536 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c3301f-7e55-4e81-b5ce-420873ef1242-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:20.508593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.508550 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:20.508593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.508567 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:20.509688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.509654 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-kube-api-access-vsbpd" pod "e6c3301f-7e55-4e81-b5ce-420873ef1242" (UID: "e6c3301f-7e55-4e81-b5ce-420873ef1242"). InnerVolumeSpecName "kube-api-access-vsbpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:20.509812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.509789 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-podinfo" pod "e6c3301f-7e55-4e81-b5ce-420873ef1242" (UID: "e6c3301f-7e55-4e81-b5ce-420873ef1242"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:20.510134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.510101 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-envoy" pod "e6c3301f-7e55-4e81-b5ce-420873ef1242" (UID: "e6c3301f-7e55-4e81-b5ce-420873ef1242"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:20.510332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.510314 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-token" pod "e6c3301f-7e55-4e81-b5ce-420873ef1242" (UID: "e6c3301f-7e55-4e81-b5ce-420873ef1242"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:20.609981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.609947 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:20.609981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.609978 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:20.609981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.609990 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c3301f-7e55-4e81-b5ce-420873ef1242-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:20.610220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.609999 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsbpd\" (UniqueName: \"kubernetes.io/projected/e6c3301f-7e55-4e81-b5ce-420873ef1242-kube-api-access-vsbpd\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:20.799691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.799658 4298 generic.go:356] "Generic (PLEG): container finished" podID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerID="37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979" exitCode=0 Nov 03 22:51:20.799882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.799716 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" Nov 03 22:51:20.799882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.799742 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" event={"ID":"e6c3301f-7e55-4e81-b5ce-420873ef1242","Type":"ContainerDied","Data":"37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979"} Nov 03 22:51:20.799882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.799802 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb" event={"ID":"e6c3301f-7e55-4e81-b5ce-420873ef1242","Type":"ContainerDied","Data":"90c0050bad3a647595f6a099be3e00fb073742650e3f859a167fb6fc6eced462"} Nov 03 22:51:20.799882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.799825 4298 scope.go:122] "RemoveContainer" containerID="37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979" Nov 03 22:51:20.810816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.810769 4298 scope.go:122] "RemoveContainer" containerID="5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726" Nov 03 22:51:20.820067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.820046 4298 scope.go:122] "RemoveContainer" containerID="37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979" Nov 03 22:51:20.820305 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:51:20.820277 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979\": container with ID starting with 37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979 not found: ID does not exist" containerID="37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979" Nov 03 22:51:20.820431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.820313 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979"} err="failed to get container status \"37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979\": rpc error: code = NotFound desc = could not find container \"37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979\": container with ID starting with 37cf87934221b79c6d2274b0a8b60523130727a40be67116f47f33421a22a979 not found: ID does not exist" Nov 03 22:51:20.820431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.820336 4298 scope.go:122] "RemoveContainer" containerID="5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726" Nov 03 22:51:20.820612 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:51:20.820593 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726\": container with ID starting with 5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726 not found: ID does not exist" containerID="5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726" Nov 03 22:51:20.820673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.820618 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726"} err="failed to get container status \"5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726\": rpc error: code = NotFound desc = could not find container \"5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726\": container with ID starting with 5cb03060bb4fa144045867beb9dd57cb97015c58422e398f5dc63f49d9a3a726 not found: ID does not exist" Nov 03 22:51:20.970883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:20.970845 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb"] Nov 03 22:51:21.021039 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:21.021011 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-97965e-openshift-default-998c88b7f-rgwmb"] Nov 03 22:51:21.136139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:21.136065 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89988bb-6b60-424a-b35e-1e889fc9f9ea" path="/var/lib/kubelet/pods/a89988bb-6b60-424a-b35e-1e889fc9f9ea/volumes" Nov 03 22:51:21.136532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:21.136518 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" path="/var/lib/kubelet/pods/e6c3301f-7e55-4e81-b5ce-420873ef1242/volumes" Nov 03 22:51:21.808566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:21.808534 4298 generic.go:356] "Generic (PLEG): container finished" podID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerID="24318ed91e9a2ac3b4f2fa5d1d2efd3736deeb49f338e55bd65ab31c4be98052" exitCode=0 Nov 03 22:51:21.808994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:21.808602 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" event={"ID":"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2","Type":"ContainerDied","Data":"24318ed91e9a2ac3b4f2fa5d1d2efd3736deeb49f338e55bd65ab31c4be98052"} Nov 03 22:51:21.808994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:21.808698 4298 scope.go:122] "RemoveContainer" containerID="3a27aaef636276f008c93be040595d260221e3fe8ebb0ce1e70b9d80f15a712d" Nov 03 22:51:21.937154 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:21.937128 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:51:22.028074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028043 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-podinfo\") pod \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " Nov 03 22:51:22.028273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028134 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istiod-ca-cert\") pod \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " Nov 03 22:51:22.028273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028237 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-socket\") pod \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " Nov 03 22:51:22.028273 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028269 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-kube-api-access-q4hsb\" (UniqueName: \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-kube-api-access-q4hsb\") pod \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " Nov 03 22:51:22.028475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028329 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-certs\") pod \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " Nov 03 22:51:22.028475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028400 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-credential-socket\") pod \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " Nov 03 22:51:22.028475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028434 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-token\" (UniqueName: \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-token\") pod \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " Nov 03 22:51:22.028669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028501 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-data\") pod \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " Nov 03 22:51:22.028669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028504 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-socket" pod "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" (UID: "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:22.028669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028524 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istiod-ca-cert" pod "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" (UID: "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:22.028669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028535 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-envoy\") pod \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\" (UID: \"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2\") " Nov 03 22:51:22.028900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028706 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-certs" pod "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" (UID: "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:22.028900 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028726 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-credential-socket" pod "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" (UID: "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:22.029009 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.028952 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-data" pod "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" (UID: "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:22.029142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.029117 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:22.029278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.029150 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:22.029278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.029160 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:22.029278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.029169 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:22.029278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.029182 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:22.030506 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.030480 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-podinfo" pod "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" (UID: "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:22.030613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.030519 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-kube-api-access-q4hsb" pod "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" (UID: "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2"). InnerVolumeSpecName "kube-api-access-q4hsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:22.030993 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.030968 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-envoy" pod "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" (UID: "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:22.031085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.030971 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-token" pod "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" (UID: "2811ac9d-28b8-4f35-b0b0-82079bd5d3b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:22.129849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.129817 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:22.129849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.129846 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:22.129849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.129856 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4hsb\" (UniqueName: \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-kube-api-access-q4hsb\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:22.130077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.129866 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:22.434385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.434295 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7"] Nov 03 22:51:22.434565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.434538 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" containerID="cri-o://c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689" gracePeriod=30 Nov 03 22:51:22.438581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.438556 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk"] Nov 03 22:51:22.438752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.438733 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" containerID="cri-o://d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89" gracePeriod=30 Nov 03 22:51:22.484397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.484352 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm"] Nov 03 22:51:22.484634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.484560 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" containerID="cri-o://b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a" gracePeriod=30 Nov 03 22:51:22.817473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.817443 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" Nov 03 22:51:22.817935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.817471 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv" event={"ID":"2811ac9d-28b8-4f35-b0b0-82079bd5d3b2","Type":"ContainerDied","Data":"22f654dc8f8c15744b2f6af01c2318623ce1d53fa709b728a61a54ca7c437a35"} Nov 03 22:51:22.817935 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.817532 4298 scope.go:122] "RemoveContainer" containerID="24318ed91e9a2ac3b4f2fa5d1d2efd3736deeb49f338e55bd65ab31c4be98052" Nov 03 22:51:22.889336 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.889302 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:22.986248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.986212 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv"] Nov 03 22:51:22.990163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:22.990137 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-fg5hv"] Nov 03 22:51:23.134745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:23.134675 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" path="/var/lib/kubelet/pods/2811ac9d-28b8-4f35-b0b0-82079bd5d3b2/volumes" Nov 03 22:51:24.906042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:24.906005 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:26.705265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:26.705221 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:26.969472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:26.969390 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:27.054229 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.054200 4298 scope.go:122] "RemoveContainer" containerID="933548b571dc3d066e34c3541bce57c66b333155d43a01a518ac36093a92cc4a" Nov 03 22:51:27.063853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.063833 4298 scope.go:122] "RemoveContainer" containerID="690077cbadb6d0e756cbc3b8b0579d65b6f43e724c8267d703c12fa84e6a16ae" Nov 03 22:51:27.072117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.072099 4298 scope.go:122] "RemoveContainer" containerID="484b99b91e6f3cd8a244885e6a03e68a07950f323f33a5b16e037d17cc8da45f" Nov 03 22:51:27.080281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.080259 4298 scope.go:122] "RemoveContainer" containerID="36ecc8902915c49212593186367ed1c3de167ca42085b757bd760e1ada1ce8cc" Nov 03 22:51:27.190346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.190308 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:27.675324 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.675230 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:27.749416 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.749388 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:51:27.771407 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.771381 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:51:27.799688 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.799658 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-data\") pod \"edb5598d-15f2-4625-ad51-c65350e4a159\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " Nov 03 22:51:27.799851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.799708 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb5598d-15f2-4625-ad51-c65350e4a159-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb5598d-15f2-4625-ad51-c65350e4a159-istiod-ca-cert\") pod \"edb5598d-15f2-4625-ad51-c65350e4a159\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " Nov 03 22:51:27.799851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.799812 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-token\" (UniqueName: \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-token\") pod \"b506bf8c-2001-4044-9e2e-b18b427cb33b\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " Nov 03 22:51:27.799972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.799917 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-data" pod "edb5598d-15f2-4625-ad51-c65350e4a159" (UID: "edb5598d-15f2-4625-ad51-c65350e4a159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.799972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.799948 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-kube-api-access-c82nt\" (UniqueName: \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-kube-api-access-c82nt\") pod \"edb5598d-15f2-4625-ad51-c65350e4a159\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " Nov 03 22:51:27.800090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.799986 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-socket\") pod \"b506bf8c-2001-4044-9e2e-b18b427cb33b\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " Nov 03 22:51:27.800090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800037 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-credential-socket\") pod \"edb5598d-15f2-4625-ad51-c65350e4a159\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " Nov 03 22:51:27.800090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800076 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-podinfo\") pod \"b506bf8c-2001-4044-9e2e-b18b427cb33b\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " Nov 03 22:51:27.800249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800109 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-envoy\") pod \"b506bf8c-2001-4044-9e2e-b18b427cb33b\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " Nov 03 22:51:27.800249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800160 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb5598d-15f2-4625-ad51-c65350e4a159-istiod-ca-cert" pod "edb5598d-15f2-4625-ad51-c65350e4a159" (UID: "edb5598d-15f2-4625-ad51-c65350e4a159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:27.800249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800202 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-credential-socket\") pod \"b506bf8c-2001-4044-9e2e-b18b427cb33b\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " Nov 03 22:51:27.800412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800246 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-certs\") pod \"b506bf8c-2001-4044-9e2e-b18b427cb33b\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " Nov 03 22:51:27.800412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800320 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-socket\") pod \"edb5598d-15f2-4625-ad51-c65350e4a159\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " Nov 03 22:51:27.800412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800349 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-kube-api-access-hwr82\" (UniqueName: \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-kube-api-access-hwr82\") pod \"b506bf8c-2001-4044-9e2e-b18b427cb33b\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " Nov 03 22:51:27.800412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800389 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edb5598d-15f2-4625-ad51-c65350e4a159-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb5598d-15f2-4625-ad51-c65350e4a159-istio-podinfo\") pod \"edb5598d-15f2-4625-ad51-c65350e4a159\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " Nov 03 22:51:27.800630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800416 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-istio-token\" (UniqueName: \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-istio-token\") pod \"edb5598d-15f2-4625-ad51-c65350e4a159\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " Nov 03 22:51:27.800630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800453 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-data\") pod \"b506bf8c-2001-4044-9e2e-b18b427cb33b\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " Nov 03 22:51:27.800630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800496 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-envoy\") pod \"edb5598d-15f2-4625-ad51-c65350e4a159\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " Nov 03 22:51:27.800630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800466 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-socket" pod "b506bf8c-2001-4044-9e2e-b18b427cb33b" (UID: "b506bf8c-2001-4044-9e2e-b18b427cb33b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.800630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800535 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-certs\") pod \"edb5598d-15f2-4625-ad51-c65350e4a159\" (UID: \"edb5598d-15f2-4625-ad51-c65350e4a159\") " Nov 03 22:51:27.800630 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800582 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b506bf8c-2001-4044-9e2e-b18b427cb33b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b506bf8c-2001-4044-9e2e-b18b427cb33b-istiod-ca-cert\") pod \"b506bf8c-2001-4044-9e2e-b18b427cb33b\" (UID: \"b506bf8c-2001-4044-9e2e-b18b427cb33b\") " Nov 03 22:51:27.800946 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.800901 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-socket" pod "edb5598d-15f2-4625-ad51-c65350e4a159" (UID: "edb5598d-15f2-4625-ad51-c65350e4a159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.801313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801104 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-certs" pod "b506bf8c-2001-4044-9e2e-b18b427cb33b" (UID: "b506bf8c-2001-4044-9e2e-b18b427cb33b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.801313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801229 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-data" pod "b506bf8c-2001-4044-9e2e-b18b427cb33b" (UID: "b506bf8c-2001-4044-9e2e-b18b427cb33b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.801480 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801398 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-certs" pod "edb5598d-15f2-4625-ad51-c65350e4a159" (UID: "edb5598d-15f2-4625-ad51-c65350e4a159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.801576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801549 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b506bf8c-2001-4044-9e2e-b18b427cb33b-istiod-ca-cert" pod "b506bf8c-2001-4044-9e2e-b18b427cb33b" (UID: "b506bf8c-2001-4044-9e2e-b18b427cb33b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:27.801834 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801814 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb5598d-15f2-4625-ad51-c65350e4a159-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.801904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801839 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.801904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801856 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.801904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801871 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.801904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801885 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.801904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801898 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.802138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801913 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b506bf8c-2001-4044-9e2e-b18b427cb33b-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.802138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.801928 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.802138 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.802003 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-credential-socket" pod "b506bf8c-2001-4044-9e2e-b18b427cb33b" (UID: "b506bf8c-2001-4044-9e2e-b18b427cb33b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.802312 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.802290 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-token" pod "b506bf8c-2001-4044-9e2e-b18b427cb33b" (UID: "b506bf8c-2001-4044-9e2e-b18b427cb33b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:27.802858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.802798 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-credential-socket" pod "edb5598d-15f2-4625-ad51-c65350e4a159" (UID: "edb5598d-15f2-4625-ad51-c65350e4a159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.802858 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.802825 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-kube-api-access-c82nt" pod "edb5598d-15f2-4625-ad51-c65350e4a159" (UID: "edb5598d-15f2-4625-ad51-c65350e4a159"). InnerVolumeSpecName "kube-api-access-c82nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:27.804333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.804185 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-envoy" pod "edb5598d-15f2-4625-ad51-c65350e4a159" (UID: "edb5598d-15f2-4625-ad51-c65350e4a159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.804333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.804285 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-podinfo" pod "b506bf8c-2001-4044-9e2e-b18b427cb33b" (UID: "b506bf8c-2001-4044-9e2e-b18b427cb33b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:27.804944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.804904 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-istio-token" pod "edb5598d-15f2-4625-ad51-c65350e4a159" (UID: "edb5598d-15f2-4625-ad51-c65350e4a159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:27.804944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.804914 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edb5598d-15f2-4625-ad51-c65350e4a159-istio-podinfo" pod "edb5598d-15f2-4625-ad51-c65350e4a159" (UID: "edb5598d-15f2-4625-ad51-c65350e4a159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:27.805372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.805347 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-envoy" pod "b506bf8c-2001-4044-9e2e-b18b427cb33b" (UID: "b506bf8c-2001-4044-9e2e-b18b427cb33b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.805556 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.805538 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-kube-api-access-hwr82" pod "b506bf8c-2001-4044-9e2e-b18b427cb33b" (UID: "b506bf8c-2001-4044-9e2e-b18b427cb33b"). InnerVolumeSpecName "kube-api-access-hwr82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:27.815599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.815580 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:51:27.856965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.856929 4298 generic.go:356] "Generic (PLEG): container finished" podID="edb5598d-15f2-4625-ad51-c65350e4a159" containerID="d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89" exitCode=0 Nov 03 22:51:27.857105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.856972 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" Nov 03 22:51:27.857105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.857015 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" event={"ID":"edb5598d-15f2-4625-ad51-c65350e4a159","Type":"ContainerDied","Data":"d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89"} Nov 03 22:51:27.857105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.857062 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk" event={"ID":"edb5598d-15f2-4625-ad51-c65350e4a159","Type":"ContainerDied","Data":"9f4c5148afbee13e303c19bcb4e2f5047cd03ca4c9701d42921e13d6835205d4"} Nov 03 22:51:27.857105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.857091 4298 scope.go:122] "RemoveContainer" containerID="d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89" Nov 03 22:51:27.858462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.858438 4298 generic.go:356] "Generic (PLEG): container finished" podID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerID="b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a" exitCode=0 Nov 03 22:51:27.858588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.858490 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" Nov 03 22:51:27.858588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.858522 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" event={"ID":"b8bd390f-7aef-4788-84cb-ae1ae867c6d1","Type":"ContainerDied","Data":"b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a"} Nov 03 22:51:27.858588 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.858578 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm" event={"ID":"b8bd390f-7aef-4788-84cb-ae1ae867c6d1","Type":"ContainerDied","Data":"52e33a4932c793e335685198af10758b1b74fa5a90b947ea8389841de36e0e28"} Nov 03 22:51:27.859794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.859774 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp"] Nov 03 22:51:27.860096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860079 4298 generic.go:356] "Generic (PLEG): container finished" podID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerID="c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689" exitCode=0 Nov 03 22:51:27.860200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860180 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" Nov 03 22:51:27.860431 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860414 4298 cpu_manager.go:414] "containerMap: removing container" podUID="155be8a8-7a3f-4f38-898d-1353747ca4c8" containerName="revoke-keys" Nov 03 22:51:27.860507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860432 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="155be8a8-7a3f-4f38-898d-1353747ca4c8" containerName="revoke-keys" Nov 03 22:51:27.860507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860456 4298 cpu_manager.go:414] "containerMap: removing container" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" Nov 03 22:51:27.860507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860463 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" Nov 03 22:51:27.860507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860477 4298 cpu_manager.go:414] "containerMap: removing container" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" Nov 03 22:51:27.860507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860485 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" Nov 03 22:51:27.860507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860504 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860511 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860538 4298 cpu_manager.go:414] "containerMap: removing container" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860546 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860565 4298 cpu_manager.go:414] "containerMap: removing container" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860572 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860587 4298 cpu_manager.go:414] "containerMap: removing container" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860594 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860622 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860629 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860770 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860787 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860798 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860808 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860823 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c3301f-7e55-4e81-b5ce-420873ef1242" containerName="istio-proxy" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860833 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="155be8a8-7a3f-4f38-898d-1353747ca4c8" containerName="revoke-keys" Nov 03 22:51:27.860857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.860845 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" containerName="istio-proxy" Nov 03 22:51:27.864433 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.864411 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" event={"ID":"b506bf8c-2001-4044-9e2e-b18b427cb33b","Type":"ContainerDied","Data":"c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689"} Nov 03 22:51:27.864526 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.864447 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7" event={"ID":"b506bf8c-2001-4044-9e2e-b18b427cb33b","Type":"ContainerDied","Data":"5ac1008451fbbe3986d623af96e860ce8583e1e86c719a9f553f61e0aff402c3"} Nov 03 22:51:27.864573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.864545 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:27.867561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.867538 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:27.869325 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.869310 4298 scope.go:122] "RemoveContainer" containerID="d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89" Nov 03 22:51:27.869595 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:51:27.869569 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89\": container with ID starting with d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89 not found: ID does not exist" containerID="d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89" Nov 03 22:51:27.869701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.869596 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89"} err="failed to get container status \"d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89\": rpc error: code = NotFound desc = could not find container \"d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89\": container with ID starting with d9008c1c90cbdf35f5bce9ba024f4d10f8d11046dcdc2bdabad6dbac70b7dc89 not found: ID does not exist" Nov 03 22:51:27.869701 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.869618 4298 scope.go:122] "RemoveContainer" containerID="b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a" Nov 03 22:51:27.879864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.879846 4298 scope.go:122] "RemoveContainer" containerID="b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a" Nov 03 22:51:27.880131 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:51:27.880109 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a\": container with ID starting with b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a not found: ID does not exist" containerID="b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a" Nov 03 22:51:27.880204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.880137 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a"} err="failed to get container status \"b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a\": rpc error: code = NotFound desc = could not find container \"b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a\": container with ID starting with b539149ae8f9efb8f80470f75ec910632360e073b4961c454f18edafd8f2227a not found: ID does not exist" Nov 03 22:51:27.880204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.880152 4298 scope.go:122] "RemoveContainer" containerID="c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689" Nov 03 22:51:27.890296 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.890276 4298 scope.go:122] "RemoveContainer" containerID="c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689" Nov 03 22:51:27.890608 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:51:27.890587 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689\": container with ID starting with c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689 not found: ID does not exist" containerID="c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689" Nov 03 22:51:27.890721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.890616 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689"} err="failed to get container status \"c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689\": rpc error: code = NotFound desc = could not find container \"c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689\": container with ID starting with c7d15b80877a5919620ec4d2496b139098d3a0fec46982cd9eab9c311e3cf689 not found: ID does not exist" Nov 03 22:51:27.892762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.892742 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp"] Nov 03 22:51:27.902893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.902869 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-socket\") pod \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " Nov 03 22:51:27.903003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.902949 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-podinfo\") pod \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " Nov 03 22:51:27.903267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903006 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-kube-api-access-kr7qd\" (UniqueName: \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-kube-api-access-kr7qd\") pod \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " Nov 03 22:51:27.903267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903042 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istiod-ca-cert\") pod \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " Nov 03 22:51:27.903267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903079 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-data\") pod \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " Nov 03 22:51:27.903267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903112 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-certs\") pod \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " Nov 03 22:51:27.903267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903124 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-socket" pod "b8bd390f-7aef-4788-84cb-ae1ae867c6d1" (UID: "b8bd390f-7aef-4788-84cb-ae1ae867c6d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.903267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903213 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-envoy\") pod \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " Nov 03 22:51:27.903267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903241 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-token\" (UniqueName: \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-token\") pod \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " Nov 03 22:51:27.903560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903292 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-credential-socket\") pod \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\" (UID: \"b8bd390f-7aef-4788-84cb-ae1ae867c6d1\") " Nov 03 22:51:27.903560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903383 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istiod-ca-cert" pod "b8bd390f-7aef-4788-84cb-ae1ae867c6d1" (UID: "b8bd390f-7aef-4788-84cb-ae1ae867c6d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:27.903683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903638 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-certs" pod "b8bd390f-7aef-4788-84cb-ae1ae867c6d1" (UID: "b8bd390f-7aef-4788-84cb-ae1ae867c6d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.903764 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903743 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:27.903899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903877 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-credential-socket" pod "b8bd390f-7aef-4788-84cb-ae1ae867c6d1" (UID: "b8bd390f-7aef-4788-84cb-ae1ae867c6d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.904023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.903968 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:27.904023 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904014 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn26f\" (UniqueName: \"kubernetes.io/projected/b39aaecd-14df-4068-b54a-bec762e4a528-kube-api-access-vn26f\") pod \"maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:27.904149 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904017 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-data" pod "b8bd390f-7aef-4788-84cb-ae1ae867c6d1" (UID: "b8bd390f-7aef-4788-84cb-ae1ae867c6d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.904270 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904246 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904277 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904292 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c82nt\" (UniqueName: \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-kube-api-access-c82nt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904307 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904320 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904334 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904346 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904362 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904385 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904399 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b506bf8c-2001-4044-9e2e-b18b427cb33b-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904412 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904426 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwr82\" (UniqueName: \"kubernetes.io/projected/b506bf8c-2001-4044-9e2e-b18b427cb33b-kube-api-access-hwr82\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904440 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb5598d-15f2-4625-ad51-c65350e4a159-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904453 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb5598d-15f2-4625-ad51-c65350e4a159-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.904620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.904466 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb5598d-15f2-4625-ad51-c65350e4a159-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:27.905576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.905552 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-podinfo" pod "b8bd390f-7aef-4788-84cb-ae1ae867c6d1" (UID: "b8bd390f-7aef-4788-84cb-ae1ae867c6d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:27.905747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.905730 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-kube-api-access-kr7qd" pod "b8bd390f-7aef-4788-84cb-ae1ae867c6d1" (UID: "b8bd390f-7aef-4788-84cb-ae1ae867c6d1"). InnerVolumeSpecName "kube-api-access-kr7qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:27.906423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.906405 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-token" pod "b8bd390f-7aef-4788-84cb-ae1ae867c6d1" (UID: "b8bd390f-7aef-4788-84cb-ae1ae867c6d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:27.906812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.906796 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-envoy" pod "b8bd390f-7aef-4788-84cb-ae1ae867c6d1" (UID: "b8bd390f-7aef-4788-84cb-ae1ae867c6d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:27.937095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.937064 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:51:27.937266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:27.937163 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:51:28.005950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.005916 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:28.005950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.005957 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn26f\" (UniqueName: \"kubernetes.io/projected/b39aaecd-14df-4068-b54a-bec762e4a528-kube-api-access-vn26f\") pod \"maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:28.006177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.006041 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:28.006177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.006083 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:28.006177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.006095 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:28.006177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.006104 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:28.006177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.006113 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr7qd\" (UniqueName: \"kubernetes.io/projected/b8bd390f-7aef-4788-84cb-ae1ae867c6d1-kube-api-access-kr7qd\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:28.006779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.006757 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:28.008333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.008312 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:28.052302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.052272 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk"] Nov 03 22:51:28.057950 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.057926 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ngvlk"] Nov 03 22:51:28.062966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.062944 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn26f\" (UniqueName: \"kubernetes.io/projected/b39aaecd-14df-4068-b54a-bec762e4a528-kube-api-access-vn26f\") pod \"maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:28.184638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.184610 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:28.348716 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.348675 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7"] Nov 03 22:51:28.352562 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.352537 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zkjl7"] Nov 03 22:51:28.371676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.371633 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:28.555909 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:51:28.555873 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39aaecd_14df_4068_b54a_bec762e4a528.slice/crio-95a4cdd84c92663272721bb8583f2e5bbf8d630e64b2a3cd296a14762d2a8442 WatchSource:0}: Error finding container 95a4cdd84c92663272721bb8583f2e5bbf8d630e64b2a3cd296a14762d2a8442: Status 404 returned error can't find the container with id 95a4cdd84c92663272721bb8583f2e5bbf8d630e64b2a3cd296a14762d2a8442 Nov 03 22:51:28.560604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.560583 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp"] Nov 03 22:51:28.582604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.582576 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm"] Nov 03 22:51:28.586519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.586497 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sn4pm"] Nov 03 22:51:28.869167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.869134 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" event={"ID":"b39aaecd-14df-4068-b54a-bec762e4a528","Type":"ContainerStarted","Data":"d57b4e5c6e14ad9fc8cd549ba0bccd3922c23f503ffc0799299ecb653be44469"} Nov 03 22:51:28.869576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.869191 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" event={"ID":"b39aaecd-14df-4068-b54a-bec762e4a528","Type":"ContainerStarted","Data":"95a4cdd84c92663272721bb8583f2e5bbf8d630e64b2a3cd296a14762d2a8442"} Nov 03 22:51:28.941514 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.941483 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm"] Nov 03 22:51:28.987458 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:28.987354 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" podStartSLOduration=1.98733855 podStartE2EDuration="1.98733855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:28.983039394 +0000 UTC m=+2645.963773044" watchObservedRunningTime="2026-11-03 22:51:28.98733855 +0000 UTC m=+2645.968072202" Nov 03 22:51:29.049813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:29.049777 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj"] Nov 03 22:51:29.136034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:29.135995 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b506bf8c-2001-4044-9e2e-b18b427cb33b" path="/var/lib/kubelet/pods/b506bf8c-2001-4044-9e2e-b18b427cb33b/volumes" Nov 03 22:51:29.136570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:29.136548 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8bd390f-7aef-4788-84cb-ae1ae867c6d1" path="/var/lib/kubelet/pods/b8bd390f-7aef-4788-84cb-ae1ae867c6d1/volumes" Nov 03 22:51:29.137052 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:29.137033 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb5598d-15f2-4625-ad51-c65350e4a159" path="/var/lib/kubelet/pods/edb5598d-15f2-4625-ad51-c65350e4a159/volumes" Nov 03 22:51:29.357234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:29.356316 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:29.554351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:29.554317 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:29.877607 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:29.877573 4298 generic.go:356] "Generic (PLEG): container finished" podID="b39aaecd-14df-4068-b54a-bec762e4a528" containerID="d57b4e5c6e14ad9fc8cd549ba0bccd3922c23f503ffc0799299ecb653be44469" exitCode=0 Nov 03 22:51:29.878044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:29.877674 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" event={"ID":"b39aaecd-14df-4068-b54a-bec762e4a528","Type":"ContainerDied","Data":"d57b4e5c6e14ad9fc8cd549ba0bccd3922c23f503ffc0799299ecb653be44469"} Nov 03 22:51:30.275585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:30.275545 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:31.074816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.074792 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:31.163452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.163378 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-openshift-ca\") pod \"b39aaecd-14df-4068-b54a-bec762e4a528\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " Nov 03 22:51:31.163452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.163409 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39aaecd-14df-4068-b54a-bec762e4a528-kube-api-access-vn26f\" (UniqueName: \"kubernetes.io/projected/b39aaecd-14df-4068-b54a-bec762e4a528-kube-api-access-vn26f\") pod \"b39aaecd-14df-4068-b54a-bec762e4a528\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " Nov 03 22:51:31.163713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.163454 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-xks-ca\") pod \"b39aaecd-14df-4068-b54a-bec762e4a528\" (UID: \"b39aaecd-14df-4068-b54a-bec762e4a528\") " Nov 03 22:51:31.163817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.163791 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-openshift-ca" pod "b39aaecd-14df-4068-b54a-bec762e4a528" (UID: "b39aaecd-14df-4068-b54a-bec762e4a528"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:31.163988 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.163971 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:31.165504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.165482 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-xks-ca" pod "b39aaecd-14df-4068-b54a-bec762e4a528" (UID: "b39aaecd-14df-4068-b54a-bec762e4a528"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:31.165635 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.165619 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39aaecd-14df-4068-b54a-bec762e4a528-kube-api-access-vn26f" pod "b39aaecd-14df-4068-b54a-bec762e4a528" (UID: "b39aaecd-14df-4068-b54a-bec762e4a528"). InnerVolumeSpecName "kube-api-access-vn26f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:31.264823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.264785 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn26f\" (UniqueName: \"kubernetes.io/projected/b39aaecd-14df-4068-b54a-bec762e4a528-kube-api-access-vn26f\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:31.264823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.264823 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39aaecd-14df-4068-b54a-bec762e4a528-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:31.892230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.892200 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" Nov 03 22:51:31.892230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.892211 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp" event={"ID":"b39aaecd-14df-4068-b54a-bec762e4a528","Type":"ContainerDied","Data":"95a4cdd84c92663272721bb8583f2e5bbf8d630e64b2a3cd296a14762d2a8442"} Nov 03 22:51:31.892469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:31.892259 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95a4cdd84c92663272721bb8583f2e5bbf8d630e64b2a3cd296a14762d2a8442" Nov 03 22:51:33.941845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:33.941756 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" podUID="6ed647c8-3abf-4243-a464-cf1fbbef7d4a" containerName="payload-processing" containerID="cri-o://75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252" gracePeriod=25 Nov 03 22:51:34.050201 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.050142 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" podUID="4f935ec2-2164-46e1-afe2-68fe6c5b4f24" containerName="payload-pre-processing" containerID="cri-o://b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d" gracePeriod=25 Nov 03 22:51:34.214795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.214763 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:51:34.298305 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.298274 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-plugins-config-volume\") pod \"6ed647c8-3abf-4243-a464-cf1fbbef7d4a\" (UID: \"6ed647c8-3abf-4243-a464-cf1fbbef7d4a\") " Nov 03 22:51:34.298478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.298388 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-kube-api-access-n6zcs\" (UniqueName: \"kubernetes.io/projected/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-kube-api-access-n6zcs\") pod \"6ed647c8-3abf-4243-a464-cf1fbbef7d4a\" (UID: \"6ed647c8-3abf-4243-a464-cf1fbbef7d4a\") " Nov 03 22:51:34.298692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.298665 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-plugins-config-volume" pod "6ed647c8-3abf-4243-a464-cf1fbbef7d4a" (UID: "6ed647c8-3abf-4243-a464-cf1fbbef7d4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:34.300563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.300540 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-kube-api-access-n6zcs" pod "6ed647c8-3abf-4243-a464-cf1fbbef7d4a" (UID: "6ed647c8-3abf-4243-a464-cf1fbbef7d4a"). InnerVolumeSpecName "kube-api-access-n6zcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:34.321237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.321216 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:51:34.399175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.399145 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-kube-api-access-f274s\" (UniqueName: \"kubernetes.io/projected/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-kube-api-access-f274s\") pod \"4f935ec2-2164-46e1-afe2-68fe6c5b4f24\" (UID: \"4f935ec2-2164-46e1-afe2-68fe6c5b4f24\") " Nov 03 22:51:34.399338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.399323 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-plugins-config-volume\") pod \"4f935ec2-2164-46e1-afe2-68fe6c5b4f24\" (UID: \"4f935ec2-2164-46e1-afe2-68fe6c5b4f24\") " Nov 03 22:51:34.399762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.399737 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-plugins-config-volume" pod "4f935ec2-2164-46e1-afe2-68fe6c5b4f24" (UID: "4f935ec2-2164-46e1-afe2-68fe6c5b4f24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:34.399915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.399900 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:34.399966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.399918 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:34.399966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.399927 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6zcs\" (UniqueName: \"kubernetes.io/projected/6ed647c8-3abf-4243-a464-cf1fbbef7d4a-kube-api-access-n6zcs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:34.401205 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.401175 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-kube-api-access-f274s" pod "4f935ec2-2164-46e1-afe2-68fe6c5b4f24" (UID: "4f935ec2-2164-46e1-afe2-68fe6c5b4f24"). InnerVolumeSpecName "kube-api-access-f274s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:34.501237 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.501199 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f274s\" (UniqueName: \"kubernetes.io/projected/4f935ec2-2164-46e1-afe2-68fe6c5b4f24-kube-api-access-f274s\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:34.912561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.912488 4298 generic.go:356] "Generic (PLEG): container finished" podID="6ed647c8-3abf-4243-a464-cf1fbbef7d4a" containerID="75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252" exitCode=0 Nov 03 22:51:34.912561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.912530 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" Nov 03 22:51:34.912814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.912565 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" event={"ID":"6ed647c8-3abf-4243-a464-cf1fbbef7d4a","Type":"ContainerDied","Data":"75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252"} Nov 03 22:51:34.912814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.912621 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm" event={"ID":"6ed647c8-3abf-4243-a464-cf1fbbef7d4a","Type":"ContainerDied","Data":"ec6515f02dbd80ee92ac97172b02a9ef790ebbb585b8abd42effeef403ecddc9"} Nov 03 22:51:34.912814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.912668 4298 scope.go:122] "RemoveContainer" containerID="75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252" Nov 03 22:51:34.914053 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.914032 4298 generic.go:356] "Generic (PLEG): container finished" podID="4f935ec2-2164-46e1-afe2-68fe6c5b4f24" containerID="b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d" exitCode=0 Nov 03 22:51:34.914152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.914077 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" Nov 03 22:51:34.914152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.914089 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" event={"ID":"4f935ec2-2164-46e1-afe2-68fe6c5b4f24","Type":"ContainerDied","Data":"b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d"} Nov 03 22:51:34.914152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.914123 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj" event={"ID":"4f935ec2-2164-46e1-afe2-68fe6c5b4f24","Type":"ContainerDied","Data":"d36cb49e37cb1bc652c0bf58dadf40fc6944c4e77fe09f76bddf25d81a6753f1"} Nov 03 22:51:34.925360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.925342 4298 scope.go:122] "RemoveContainer" containerID="75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252" Nov 03 22:51:34.925622 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:51:34.925603 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252\": container with ID starting with 75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252 not found: ID does not exist" containerID="75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252" Nov 03 22:51:34.925694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.925630 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252"} err="failed to get container status \"75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252\": rpc error: code = NotFound desc = could not find container \"75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252\": container with ID starting with 75d9f4eaa263293c0cc700b0e4953d87a8734a4d27f0528670934a0bb48da252 not found: ID does not exist" Nov 03 22:51:34.925694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.925672 4298 scope.go:122] "RemoveContainer" containerID="b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d" Nov 03 22:51:34.934938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.934923 4298 scope.go:122] "RemoveContainer" containerID="b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d" Nov 03 22:51:34.935222 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:51:34.935205 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d\": container with ID starting with b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d not found: ID does not exist" containerID="b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d" Nov 03 22:51:34.935295 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:34.935231 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d"} err="failed to get container status \"b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d\": rpc error: code = NotFound desc = could not find container \"b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d\": container with ID starting with b333303bbf0e9070268f0ab1249f34bdaba0bc374af68cae518dd38dd0e8ac1d not found: ID does not exist" Nov 03 22:51:35.080180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:35.080151 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm"] Nov 03 22:51:35.138419 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:35.138393 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-472b46-ffb9c9d-5v2bm"] Nov 03 22:51:35.193953 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:35.193870 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj"] Nov 03 22:51:35.197717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:35.197696 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472b46-597b49996c-9msjj"] Nov 03 22:51:35.512616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:35.512571 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:37.135417 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:37.135386 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f935ec2-2164-46e1-afe2-68fe6c5b4f24" path="/var/lib/kubelet/pods/4f935ec2-2164-46e1-afe2-68fe6c5b4f24/volumes" Nov 03 22:51:37.135844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:37.135827 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed647c8-3abf-4243-a464-cf1fbbef7d4a" path="/var/lib/kubelet/pods/6ed647c8-3abf-4243-a464-cf1fbbef7d4a/volumes" Nov 03 22:51:39.354496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:39.354459 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx"] Nov 03 22:51:39.762907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:39.762871 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:39.947541 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:39.947479 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:41.403693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:41.403636 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:41.618267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:41.618234 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:41.806705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:41.806667 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:41.887233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:41.887176 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr"] Nov 03 22:51:41.937219 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:41.937187 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bcd0b-h4zqr"] Nov 03 22:51:42.363634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:42.363598 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:43.135236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:43.135202 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e" path="/var/lib/kubelet/pods/c0ef8b45-ef45-4a9f-8bb9-7027e3c7ea1e/volumes" Nov 03 22:51:44.355465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:44.355426 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" podUID="c6639a62-b480-40e7-8c87-9aa9ba85160d" containerName="payload-processing" containerID="cri-o://e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33" gracePeriod=25 Nov 03 22:51:46.707300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:46.707265 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:47.553029 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:51:47.552994 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:51:47.629696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:47.629655 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" podUID="c6639a62-b480-40e7-8c87-9aa9ba85160d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:51:48.142441 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:48.142408 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:48.336487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:48.336451 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:48.447811 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:48.447778 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d"] Nov 03 22:51:48.448077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:48.448038 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" containerID="cri-o://8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a" gracePeriod=30 Nov 03 22:51:48.540241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:48.540201 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:49.000360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.000327 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:49.456631 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.456598 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7"] Nov 03 22:51:49.457105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.457091 4298 cpu_manager.go:414] "containerMap: removing container" podUID="b39aaecd-14df-4068-b54a-bec762e4a528" containerName="revoke-keys" Nov 03 22:51:49.457150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.457106 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39aaecd-14df-4068-b54a-bec762e4a528" containerName="revoke-keys" Nov 03 22:51:49.457150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.457116 4298 cpu_manager.go:414] "containerMap: removing container" podUID="6ed647c8-3abf-4243-a464-cf1fbbef7d4a" containerName="payload-processing" Nov 03 22:51:49.457150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.457121 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed647c8-3abf-4243-a464-cf1fbbef7d4a" containerName="payload-processing" Nov 03 22:51:49.457150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.457133 4298 cpu_manager.go:414] "containerMap: removing container" podUID="4f935ec2-2164-46e1-afe2-68fe6c5b4f24" containerName="payload-pre-processing" Nov 03 22:51:49.457150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.457139 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f935ec2-2164-46e1-afe2-68fe6c5b4f24" containerName="payload-pre-processing" Nov 03 22:51:49.457319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.457220 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed647c8-3abf-4243-a464-cf1fbbef7d4a" containerName="payload-processing" Nov 03 22:51:49.457319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.457231 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f935ec2-2164-46e1-afe2-68fe6c5b4f24" containerName="payload-pre-processing" Nov 03 22:51:49.457319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.457241 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="2811ac9d-28b8-4f35-b0b0-82079bd5d3b2" containerName="istio-proxy" Nov 03 22:51:49.457319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.457253 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39aaecd-14df-4068-b54a-bec762e4a528" containerName="revoke-keys" Nov 03 22:51:49.460452 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.460435 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:49.513379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.513349 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:51:49.563632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.563598 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:51:49.565166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.565139 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:49.565289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.565269 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2fd\" (UniqueName: \"kubernetes.io/projected/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-kube-api-access-6d2fd\") pod \"maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:49.565363 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.565348 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:49.567957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.567936 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7"] Nov 03 22:51:49.666281 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.666245 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:49.666447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.666346 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2fd\" (UniqueName: \"kubernetes.io/projected/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-kube-api-access-6d2fd\") pod \"maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:49.666447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.666434 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:49.667063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.667043 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:49.668664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.668633 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:49.724125 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.724045 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2fd\" (UniqueName: \"kubernetes.io/projected/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-kube-api-access-6d2fd\") pod \"maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:49.773178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:49.773145 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:50.149957 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:51:50.149926 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1ea316_d7d5_467f_b99c_8b3d8f15bcae.slice/crio-c906c876c98ef5c7bcf69447f4f3052891c39775b0f856a1ce3ad7edc1ed03c7 WatchSource:0}: Error finding container c906c876c98ef5c7bcf69447f4f3052891c39775b0f856a1ce3ad7edc1ed03c7: Status 404 returned error can't find the container with id c906c876c98ef5c7bcf69447f4f3052891c39775b0f856a1ce3ad7edc1ed03c7 Nov 03 22:51:50.153633 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:50.153608 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7"] Nov 03 22:51:50.874224 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:50.874192 4298 patch_prober.go:29] interesting pod/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:51:50.874675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:50.874260 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:51:51.022845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:51.022815 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" event={"ID":"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae","Type":"ContainerStarted","Data":"42b8fd64d2505b368feffa7e6b20497cc25671bbfd1e22405cdd75fe678fde56"} Nov 03 22:51:51.023031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:51.022863 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" event={"ID":"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae","Type":"ContainerStarted","Data":"c906c876c98ef5c7bcf69447f4f3052891c39775b0f856a1ce3ad7edc1ed03c7"} Nov 03 22:51:51.140804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:51.140694 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" podStartSLOduration=2.140680156 podStartE2EDuration="2.140680156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:51.137075648 +0000 UTC m=+2668.117809302" watchObservedRunningTime="2026-11-03 22:51:51.140680156 +0000 UTC m=+2668.121413806" Nov 03 22:51:51.257322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:51.257287 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:52.030754 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:52.030715 4298 generic.go:356] "Generic (PLEG): container finished" podID="ff1ea316-d7d5-467f-b99c-8b3d8f15bcae" containerID="42b8fd64d2505b368feffa7e6b20497cc25671bbfd1e22405cdd75fe678fde56" exitCode=0 Nov 03 22:51:52.031170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:52.030763 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" event={"ID":"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae","Type":"ContainerDied","Data":"42b8fd64d2505b368feffa7e6b20497cc25671bbfd1e22405cdd75fe678fde56"} Nov 03 22:51:52.932476 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:52.932438 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:53.182112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.182083 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:53.211655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.211610 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:53.310532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.310500 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-openshift-ca\") pod \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " Nov 03 22:51:53.310758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.310547 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-kube-api-access-6d2fd\" (UniqueName: \"kubernetes.io/projected/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-kube-api-access-6d2fd\") pod \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " Nov 03 22:51:53.310812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.310766 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-xks-ca\") pod \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\" (UID: \"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae\") " Nov 03 22:51:53.310982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.310960 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-openshift-ca" pod "ff1ea316-d7d5-467f-b99c-8b3d8f15bcae" (UID: "ff1ea316-d7d5-467f-b99c-8b3d8f15bcae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:53.311251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.311233 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.312805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.312778 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-xks-ca" pod "ff1ea316-d7d5-467f-b99c-8b3d8f15bcae" (UID: "ff1ea316-d7d5-467f-b99c-8b3d8f15bcae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:53.312906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.312862 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-kube-api-access-6d2fd" pod "ff1ea316-d7d5-467f-b99c-8b3d8f15bcae" (UID: "ff1ea316-d7d5-467f-b99c-8b3d8f15bcae"). InnerVolumeSpecName "kube-api-access-6d2fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:53.393460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.393431 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:53.402170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.402128 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f"] Nov 03 22:51:53.406353 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.406326 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-295682-v827f"] Nov 03 22:51:53.412058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.412025 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.412058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.412059 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d2fd\" (UniqueName: \"kubernetes.io/projected/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae-kube-api-access-6d2fd\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.681364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.681341 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:51:53.715381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.715308 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-istio-token\" (UniqueName: \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-istio-token\") pod \"fe800041-c967-421b-a414-2a6aebb1d69c\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " Nov 03 22:51:53.715381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.715354 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-certs\") pod \"fe800041-c967-421b-a414-2a6aebb1d69c\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " Nov 03 22:51:53.715620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.715423 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-kube-api-access-5k6tb\" (UniqueName: \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-kube-api-access-5k6tb\") pod \"fe800041-c967-421b-a414-2a6aebb1d69c\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " Nov 03 22:51:53.715620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.715503 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-data\") pod \"fe800041-c967-421b-a414-2a6aebb1d69c\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " Nov 03 22:51:53.715620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.715581 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-socket\") pod \"fe800041-c967-421b-a414-2a6aebb1d69c\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " Nov 03 22:51:53.715840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.715674 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe800041-c967-421b-a414-2a6aebb1d69c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe800041-c967-421b-a414-2a6aebb1d69c-istio-podinfo\") pod \"fe800041-c967-421b-a414-2a6aebb1d69c\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " Nov 03 22:51:53.715840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.715708 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-credential-socket\") pod \"fe800041-c967-421b-a414-2a6aebb1d69c\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " Nov 03 22:51:53.715840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.715743 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-envoy\") pod \"fe800041-c967-421b-a414-2a6aebb1d69c\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " Nov 03 22:51:53.715840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.715750 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-certs" pod "fe800041-c967-421b-a414-2a6aebb1d69c" (UID: "fe800041-c967-421b-a414-2a6aebb1d69c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:53.716044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.715944 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe800041-c967-421b-a414-2a6aebb1d69c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe800041-c967-421b-a414-2a6aebb1d69c-istiod-ca-cert\") pod \"fe800041-c967-421b-a414-2a6aebb1d69c\" (UID: \"fe800041-c967-421b-a414-2a6aebb1d69c\") " Nov 03 22:51:53.716368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.716234 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-socket" pod "fe800041-c967-421b-a414-2a6aebb1d69c" (UID: "fe800041-c967-421b-a414-2a6aebb1d69c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:53.716837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.716520 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.716837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.716544 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.716837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.716764 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-credential-socket" pod "fe800041-c967-421b-a414-2a6aebb1d69c" (UID: "fe800041-c967-421b-a414-2a6aebb1d69c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:53.716837 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.716784 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-data" pod "fe800041-c967-421b-a414-2a6aebb1d69c" (UID: "fe800041-c967-421b-a414-2a6aebb1d69c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:53.717135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.716938 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe800041-c967-421b-a414-2a6aebb1d69c-istiod-ca-cert" pod "fe800041-c967-421b-a414-2a6aebb1d69c" (UID: "fe800041-c967-421b-a414-2a6aebb1d69c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:53.718288 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.718236 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-kube-api-access-5k6tb" pod "fe800041-c967-421b-a414-2a6aebb1d69c" (UID: "fe800041-c967-421b-a414-2a6aebb1d69c"). InnerVolumeSpecName "kube-api-access-5k6tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:53.718288 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.718235 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-istio-token" pod "fe800041-c967-421b-a414-2a6aebb1d69c" (UID: "fe800041-c967-421b-a414-2a6aebb1d69c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:53.718677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.718633 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-envoy" pod "fe800041-c967-421b-a414-2a6aebb1d69c" (UID: "fe800041-c967-421b-a414-2a6aebb1d69c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:53.718973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.718954 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe800041-c967-421b-a414-2a6aebb1d69c-istio-podinfo" pod "fe800041-c967-421b-a414-2a6aebb1d69c" (UID: "fe800041-c967-421b-a414-2a6aebb1d69c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:53.817728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.817693 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe800041-c967-421b-a414-2a6aebb1d69c-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.817728 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.817728 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.817937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.817740 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.817937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.817754 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe800041-c967-421b-a414-2a6aebb1d69c-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.817937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.817767 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.817937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.817778 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6tb\" (UniqueName: \"kubernetes.io/projected/fe800041-c967-421b-a414-2a6aebb1d69c-kube-api-access-5k6tb\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.817937 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.817790 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe800041-c967-421b-a414-2a6aebb1d69c-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:53.859947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:53.859913 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:54.045278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.045202 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" Nov 03 22:51:54.045278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.045216 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7" event={"ID":"ff1ea316-d7d5-467f-b99c-8b3d8f15bcae","Type":"ContainerDied","Data":"c906c876c98ef5c7bcf69447f4f3052891c39775b0f856a1ce3ad7edc1ed03c7"} Nov 03 22:51:54.045278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.045265 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c906c876c98ef5c7bcf69447f4f3052891c39775b0f856a1ce3ad7edc1ed03c7" Nov 03 22:51:54.047029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.047001 4298 generic.go:356] "Generic (PLEG): container finished" podID="fe800041-c967-421b-a414-2a6aebb1d69c" containerID="8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a" exitCode=0 Nov 03 22:51:54.047150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.047038 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" Nov 03 22:51:54.047150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.047048 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" event={"ID":"fe800041-c967-421b-a414-2a6aebb1d69c","Type":"ContainerDied","Data":"8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a"} Nov 03 22:51:54.047150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.047094 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d" event={"ID":"fe800041-c967-421b-a414-2a6aebb1d69c","Type":"ContainerDied","Data":"6c9ed299446d69745db5390e9d3737e0f465f528da1504989bb475a492c677fd"} Nov 03 22:51:54.047150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.047124 4298 scope.go:122] "RemoveContainer" containerID="8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a" Nov 03 22:51:54.057941 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.057923 4298 scope.go:122] "RemoveContainer" containerID="d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d" Nov 03 22:51:54.068939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.068861 4298 scope.go:122] "RemoveContainer" containerID="8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a" Nov 03 22:51:54.069366 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:51:54.069341 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a\": container with ID starting with 8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a not found: ID does not exist" containerID="8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a" Nov 03 22:51:54.069474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.069374 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a"} err="failed to get container status \"8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a\": rpc error: code = NotFound desc = could not find container \"8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a\": container with ID starting with 8ed7b5242851cf577fb1517387f9d74de0d07eabdac5c7c0817ebd2e32c14c6a not found: ID does not exist" Nov 03 22:51:54.069474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.069394 4298 scope.go:122] "RemoveContainer" containerID="d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d" Nov 03 22:51:54.069696 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:51:54.069677 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d\": container with ID starting with d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d not found: ID does not exist" containerID="d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d" Nov 03 22:51:54.069775 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.069702 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d"} err="failed to get container status \"d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d\": rpc error: code = NotFound desc = could not find container \"d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d\": container with ID starting with d8299062dab12501aad65baa5c85d7d4a58b56b3cb3c7ac65fcafc45445ccf6d not found: ID does not exist" Nov 03 22:51:54.099375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.099352 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx"] Nov 03 22:51:54.216758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.216729 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d"] Nov 03 22:51:54.224583 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.224558 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7bcd0b-openshift-default-56b8bfd47f-rmt2d"] Nov 03 22:51:54.276184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:54.276138 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:55.134397 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:55.134365 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e35c5b-1acb-434a-b4d1-51f327a2373c" path="/var/lib/kubelet/pods/39e35c5b-1acb-434a-b4d1-51f327a2373c/volumes" Nov 03 22:51:55.134823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:55.134806 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" path="/var/lib/kubelet/pods/fe800041-c967-421b-a414-2a6aebb1d69c/volumes" Nov 03 22:51:57.630440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:57.630399 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" podUID="c6639a62-b480-40e7-8c87-9aa9ba85160d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:51:57.718315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:57.718271 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:58.988783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:58.988710 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:59.338112 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:59.338034 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:59.637632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:59.637533 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:59.831189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:59.831153 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr"] Nov 03 22:51:59.877922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:51:59.877889 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f61714-fpmbr"] Nov 03 22:52:00.098957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:00.098904 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:00.288845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:00.288817 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:00.361427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:00.361334 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c"] Nov 03 22:52:00.361612 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:00.361553 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" containerID="cri-o://44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5" gracePeriod=30 Nov 03 22:52:00.502210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:00.502180 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:00.831085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:00.831054 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:01.058689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.058661 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c"] Nov 03 22:52:01.135103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.135030 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be6f1a6-26ff-4087-a29b-f4f3481bda76" path="/var/lib/kubelet/pods/1be6f1a6-26ff-4087-a29b-f4f3481bda76/volumes" Nov 03 22:52:01.167062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.167039 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8"] Nov 03 22:52:01.178215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.178185 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:01.356453 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.356404 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:01.423483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.423392 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp"] Nov 03 22:52:01.427711 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.427685 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472b46-58kxp"] Nov 03 22:52:01.540066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.540034 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:01.724060 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.724028 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:01.861990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.861956 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k"] Nov 03 22:52:01.862539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.862518 4298 cpu_manager.go:414] "containerMap: removing container" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" Nov 03 22:52:01.862660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.862540 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" Nov 03 22:52:01.862660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.862571 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ff1ea316-d7d5-467f-b99c-8b3d8f15bcae" containerName="revoke-keys" Nov 03 22:52:01.862660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.862579 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1ea316-d7d5-467f-b99c-8b3d8f15bcae" containerName="revoke-keys" Nov 03 22:52:01.862845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.862737 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" Nov 03 22:52:01.862845 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.862755 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1ea316-d7d5-467f-b99c-8b3d8f15bcae" containerName="revoke-keys" Nov 03 22:52:01.867164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.867141 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:01.898059 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.898016 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:01.920130 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.920101 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:52:01.920390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.920152 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:52:01.922741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:01.922713 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k"] Nov 03 22:52:02.032165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:02.032079 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:02.032165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:02.032127 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k98r\" (UniqueName: \"kubernetes.io/projected/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-kube-api-access-8k98r\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:02.032360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:02.032290 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:02.133221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:02.133181 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:02.133221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:02.133228 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k98r\" (UniqueName: \"kubernetes.io/projected/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-kube-api-access-8k98r\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:02.133429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:02.133321 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:02.133998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:02.133977 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:02.135673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:02.135630 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:02.190744 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:02.190708 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k98r\" (UniqueName: \"kubernetes.io/projected/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-kube-api-access-8k98r\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:02.479844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:02.479809 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:02.887436 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:52:02.887401 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbd19d3_a6c5_4036_8af3_c3cc75926b3b.slice/crio-83de84414e2cedba6a50f81aa104ca90db22725951641a4b08eda15b68155ffe WatchSource:0}: Error finding container 83de84414e2cedba6a50f81aa104ca90db22725951641a4b08eda15b68155ffe: Status 404 returned error can't find the container with id 83de84414e2cedba6a50f81aa104ca90db22725951641a4b08eda15b68155ffe Nov 03 22:52:02.891730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:02.891698 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k"] Nov 03 22:52:03.116114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:03.116080 4298 patch_prober.go:29] interesting pod/e2e-api-b-295682-openshift-default-7446766979-95n6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:52:03.116290 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:03.116140 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:52:03.117078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:03.117050 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" event={"ID":"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b","Type":"ContainerStarted","Data":"737eb7fa10f07c7b215bf9708c959c9985d96fd304f8294583f0e340a7313696"} Nov 03 22:52:03.117211 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:03.117106 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" event={"ID":"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b","Type":"ContainerStarted","Data":"83de84414e2cedba6a50f81aa104ca90db22725951641a4b08eda15b68155ffe"} Nov 03 22:52:03.135841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:03.135770 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="155be8a8-7a3f-4f38-898d-1353747ca4c8" path="/var/lib/kubelet/pods/155be8a8-7a3f-4f38-898d-1353747ca4c8/volumes" Nov 03 22:52:03.233381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:03.233320 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" podStartSLOduration=2.233305928 podStartE2EDuration="2.233305928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:03.23141233 +0000 UTC m=+2680.212145983" watchObservedRunningTime="2026-11-03 22:52:03.233305928 +0000 UTC m=+2680.214039581" Nov 03 22:52:04.125798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:04.125757 4298 generic.go:356] "Generic (PLEG): container finished" podID="bbbd19d3-a6c5-4036-8af3-c3cc75926b3b" containerID="737eb7fa10f07c7b215bf9708c959c9985d96fd304f8294583f0e340a7313696" exitCode=0 Nov 03 22:52:04.125986 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:04.125846 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" event={"ID":"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b","Type":"ContainerDied","Data":"737eb7fa10f07c7b215bf9708c959c9985d96fd304f8294583f0e340a7313696"} Nov 03 22:52:04.618864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:04.618840 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:52:04.773519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:04.773426 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6639a62-b480-40e7-8c87-9aa9ba85160d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6639a62-b480-40e7-8c87-9aa9ba85160d-plugins-config-volume\") pod \"c6639a62-b480-40e7-8c87-9aa9ba85160d\" (UID: \"c6639a62-b480-40e7-8c87-9aa9ba85160d\") " Nov 03 22:52:04.773519 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:04.773474 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6639a62-b480-40e7-8c87-9aa9ba85160d-kube-api-access-q5xbl\" (UniqueName: \"kubernetes.io/projected/c6639a62-b480-40e7-8c87-9aa9ba85160d-kube-api-access-q5xbl\") pod \"c6639a62-b480-40e7-8c87-9aa9ba85160d\" (UID: \"c6639a62-b480-40e7-8c87-9aa9ba85160d\") " Nov 03 22:52:04.773906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:04.773882 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6639a62-b480-40e7-8c87-9aa9ba85160d-plugins-config-volume" pod "c6639a62-b480-40e7-8c87-9aa9ba85160d" (UID: "c6639a62-b480-40e7-8c87-9aa9ba85160d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:04.775654 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:04.775627 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6639a62-b480-40e7-8c87-9aa9ba85160d-kube-api-access-q5xbl" pod "c6639a62-b480-40e7-8c87-9aa9ba85160d" (UID: "c6639a62-b480-40e7-8c87-9aa9ba85160d"). InnerVolumeSpecName "kube-api-access-q5xbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:04.874454 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:04.874419 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6639a62-b480-40e7-8c87-9aa9ba85160d-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:04.874454 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:04.874449 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5xbl\" (UniqueName: \"kubernetes.io/projected/c6639a62-b480-40e7-8c87-9aa9ba85160d-kube-api-access-q5xbl\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.142843 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.142740 4298 generic.go:356] "Generic (PLEG): container finished" podID="c6639a62-b480-40e7-8c87-9aa9ba85160d" containerID="e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33" exitCode=0 Nov 03 22:52:05.143067 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.143038 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" event={"ID":"c6639a62-b480-40e7-8c87-9aa9ba85160d","Type":"ContainerDied","Data":"e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33"} Nov 03 22:52:05.143139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.143096 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" event={"ID":"c6639a62-b480-40e7-8c87-9aa9ba85160d","Type":"ContainerDied","Data":"d9de2762377dac5450a90e1e63f4e4d863c10582779ad066bbfc1088dc15a69f"} Nov 03 22:52:05.143139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.143124 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx" Nov 03 22:52:05.143314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.143292 4298 scope.go:122] "RemoveContainer" containerID="e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33" Nov 03 22:52:05.156006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.155940 4298 scope.go:122] "RemoveContainer" containerID="e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33" Nov 03 22:52:05.156300 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:05.156278 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33\": container with ID starting with e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33 not found: ID does not exist" containerID="e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33" Nov 03 22:52:05.156391 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.156312 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33"} err="failed to get container status \"e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33\": rpc error: code = NotFound desc = could not find container \"e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33\": container with ID starting with e1e9befb7db390346533c745e28d51b9c77927d4a1b4faeb496c61e2f46aaf33 not found: ID does not exist" Nov 03 22:52:05.295547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.295523 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:05.313030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.313000 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx"] Nov 03 22:52:05.363310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.363283 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b30a14-6c59b8b896-sxwzx"] Nov 03 22:52:05.479938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.479907 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-openshift-ca\") pod \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " Nov 03 22:52:05.480099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.480028 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-kube-api-access-8k98r\" (UniqueName: \"kubernetes.io/projected/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-kube-api-access-8k98r\") pod \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " Nov 03 22:52:05.480099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.480080 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-xks-ca\") pod \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\" (UID: \"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b\") " Nov 03 22:52:05.480889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.480855 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-openshift-ca" pod "bbbd19d3-a6c5-4036-8af3-c3cc75926b3b" (UID: "bbbd19d3-a6c5-4036-8af3-c3cc75926b3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:05.485312 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.485280 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-kube-api-access-8k98r" pod "bbbd19d3-a6c5-4036-8af3-c3cc75926b3b" (UID: "bbbd19d3-a6c5-4036-8af3-c3cc75926b3b"). InnerVolumeSpecName "kube-api-access-8k98r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:05.486128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.486097 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-xks-ca" pod "bbbd19d3-a6c5-4036-8af3-c3cc75926b3b" (UID: "bbbd19d3-a6c5-4036-8af3-c3cc75926b3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:05.581867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.581833 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.581867 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.581865 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k98r\" (UniqueName: \"kubernetes.io/projected/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-kube-api-access-8k98r\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.582071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.581881 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.623584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.623558 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:52:05.783854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.783775 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36642f0e-de5f-4793-9398-0965f2d8ea70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36642f0e-de5f-4793-9398-0965f2d8ea70-istiod-ca-cert\") pod \"36642f0e-de5f-4793-9398-0965f2d8ea70\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " Nov 03 22:52:05.783854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.783827 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-credential-socket\") pod \"36642f0e-de5f-4793-9398-0965f2d8ea70\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " Nov 03 22:52:05.783854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.783845 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-data\") pod \"36642f0e-de5f-4793-9398-0965f2d8ea70\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " Nov 03 22:52:05.784156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.783863 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-certs\") pod \"36642f0e-de5f-4793-9398-0965f2d8ea70\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " Nov 03 22:52:05.784156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.783886 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-podinfo\") pod \"36642f0e-de5f-4793-9398-0965f2d8ea70\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " Nov 03 22:52:05.784156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.783921 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-kube-api-access-ttb9n\" (UniqueName: \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-kube-api-access-ttb9n\") pod \"36642f0e-de5f-4793-9398-0965f2d8ea70\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " Nov 03 22:52:05.784156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.783960 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-envoy\") pod \"36642f0e-de5f-4793-9398-0965f2d8ea70\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " Nov 03 22:52:05.784156 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784142 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-socket\") pod \"36642f0e-de5f-4793-9398-0965f2d8ea70\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " Nov 03 22:52:05.784429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784154 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-data" pod "36642f0e-de5f-4793-9398-0965f2d8ea70" (UID: "36642f0e-de5f-4793-9398-0965f2d8ea70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:05.784429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784168 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-credential-socket" pod "36642f0e-de5f-4793-9398-0965f2d8ea70" (UID: "36642f0e-de5f-4793-9398-0965f2d8ea70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:05.784429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784205 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-certs" pod "36642f0e-de5f-4793-9398-0965f2d8ea70" (UID: "36642f0e-de5f-4793-9398-0965f2d8ea70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:05.784429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784192 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-token\" (UniqueName: \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-token\") pod \"36642f0e-de5f-4793-9398-0965f2d8ea70\" (UID: \"36642f0e-de5f-4793-9398-0965f2d8ea70\") " Nov 03 22:52:05.784429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784312 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36642f0e-de5f-4793-9398-0965f2d8ea70-istiod-ca-cert" pod "36642f0e-de5f-4793-9398-0965f2d8ea70" (UID: "36642f0e-de5f-4793-9398-0965f2d8ea70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:05.784706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784444 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-socket" pod "36642f0e-de5f-4793-9398-0965f2d8ea70" (UID: "36642f0e-de5f-4793-9398-0965f2d8ea70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:05.784807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784784 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.784874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784814 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36642f0e-de5f-4793-9398-0965f2d8ea70-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.784874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784829 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.784874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784843 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.784874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.784855 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.786410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.786381 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-token" pod "36642f0e-de5f-4793-9398-0965f2d8ea70" (UID: "36642f0e-de5f-4793-9398-0965f2d8ea70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:05.786530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.786416 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-kube-api-access-ttb9n" pod "36642f0e-de5f-4793-9398-0965f2d8ea70" (UID: "36642f0e-de5f-4793-9398-0965f2d8ea70"). InnerVolumeSpecName "kube-api-access-ttb9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:05.786530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.786419 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-podinfo" pod "36642f0e-de5f-4793-9398-0965f2d8ea70" (UID: "36642f0e-de5f-4793-9398-0965f2d8ea70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:05.786737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.786722 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-envoy" pod "36642f0e-de5f-4793-9398-0965f2d8ea70" (UID: "36642f0e-de5f-4793-9398-0965f2d8ea70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:05.886351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.886318 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.886351 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.886352 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttb9n\" (UniqueName: \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-kube-api-access-ttb9n\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.886532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.886368 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.886532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:05.886383 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36642f0e-de5f-4793-9398-0965f2d8ea70-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:06.059152 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.059066 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" podUID="3317aadb-8f90-46e2-be55-fa9b81238900" containerName="payload-processing" containerID="cri-o://a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0" gracePeriod=25 Nov 03 22:52:06.150435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.150396 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" Nov 03 22:52:06.150435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.150421 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k" event={"ID":"bbbd19d3-a6c5-4036-8af3-c3cc75926b3b","Type":"ContainerDied","Data":"83de84414e2cedba6a50f81aa104ca90db22725951641a4b08eda15b68155ffe"} Nov 03 22:52:06.150676 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.150468 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83de84414e2cedba6a50f81aa104ca90db22725951641a4b08eda15b68155ffe" Nov 03 22:52:06.153115 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.153086 4298 generic.go:356] "Generic (PLEG): container finished" podID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerID="44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5" exitCode=0 Nov 03 22:52:06.153236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.153120 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" Nov 03 22:52:06.153236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.153163 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" event={"ID":"36642f0e-de5f-4793-9398-0965f2d8ea70","Type":"ContainerDied","Data":"44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5"} Nov 03 22:52:06.153236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.153214 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c" event={"ID":"36642f0e-de5f-4793-9398-0965f2d8ea70","Type":"ContainerDied","Data":"9eee5dc2b6b981efe5978255273df5948e27d3bda0f46a46342628c5242994b4"} Nov 03 22:52:06.153413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.153247 4298 scope.go:122] "RemoveContainer" containerID="44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5" Nov 03 22:52:06.168062 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.168022 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" podUID="1aedc98d-0cc8-492e-ad70-6b7b5a661b1f" containerName="payload-pre-processing" containerID="cri-o://375dd0b04c1997bce72541d9498111c3996816b8280f6cb8d10cd680513add7c" gracePeriod=25 Nov 03 22:52:06.171879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.171847 4298 scope.go:122] "RemoveContainer" containerID="25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294" Nov 03 22:52:06.185849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.185422 4298 scope.go:122] "RemoveContainer" containerID="44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5" Nov 03 22:52:06.185849 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:06.185822 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5\": container with ID starting with 44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5 not found: ID does not exist" containerID="44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5" Nov 03 22:52:06.185943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.185848 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5"} err="failed to get container status \"44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5\": rpc error: code = NotFound desc = could not find container \"44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5\": container with ID starting with 44d2b884a2b67250b85e3c972b9e55a21f6c868f6ce453d922ed239cb1d78cd5 not found: ID does not exist" Nov 03 22:52:06.185943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.185874 4298 scope.go:122] "RemoveContainer" containerID="25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294" Nov 03 22:52:06.186271 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:06.186238 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294\": container with ID starting with 25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294 not found: ID does not exist" containerID="25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294" Nov 03 22:52:06.186403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.186280 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294"} err="failed to get container status \"25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294\": rpc error: code = NotFound desc = could not find container \"25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294\": container with ID starting with 25211e2c79a5815f7dfab8e8f14d08635eef142c84ffb0b1d8329a98fec01294 not found: ID does not exist" Nov 03 22:52:06.298753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.298729 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:52:06.340013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.339933 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c"] Nov 03 22:52:06.390423 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.390391 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-295682-openshift-default-7446766979-95n6c"] Nov 03 22:52:06.390768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.390742 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3317aadb-8f90-46e2-be55-fa9b81238900-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3317aadb-8f90-46e2-be55-fa9b81238900-plugins-config-volume\") pod \"3317aadb-8f90-46e2-be55-fa9b81238900\" (UID: \"3317aadb-8f90-46e2-be55-fa9b81238900\") " Nov 03 22:52:06.390916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.390902 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3317aadb-8f90-46e2-be55-fa9b81238900-kube-api-access-br2qh\" (UniqueName: \"kubernetes.io/projected/3317aadb-8f90-46e2-be55-fa9b81238900-kube-api-access-br2qh\") pod \"3317aadb-8f90-46e2-be55-fa9b81238900\" (UID: \"3317aadb-8f90-46e2-be55-fa9b81238900\") " Nov 03 22:52:06.391189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.391159 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3317aadb-8f90-46e2-be55-fa9b81238900-plugins-config-volume" pod "3317aadb-8f90-46e2-be55-fa9b81238900" (UID: "3317aadb-8f90-46e2-be55-fa9b81238900"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:06.391459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.391438 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3317aadb-8f90-46e2-be55-fa9b81238900-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:06.393095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.393071 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3317aadb-8f90-46e2-be55-fa9b81238900-kube-api-access-br2qh" pod "3317aadb-8f90-46e2-be55-fa9b81238900" (UID: "3317aadb-8f90-46e2-be55-fa9b81238900"). InnerVolumeSpecName "kube-api-access-br2qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:06.471511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.471477 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:06.493099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:06.493062 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br2qh\" (UniqueName: \"kubernetes.io/projected/3317aadb-8f90-46e2-be55-fa9b81238900-kube-api-access-br2qh\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:07.134558 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:07.134528 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" path="/var/lib/kubelet/pods/36642f0e-de5f-4793-9398-0965f2d8ea70/volumes" Nov 03 22:52:07.134967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:07.134950 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6639a62-b480-40e7-8c87-9aa9ba85160d" path="/var/lib/kubelet/pods/c6639a62-b480-40e7-8c87-9aa9ba85160d/volumes" Nov 03 22:52:07.161090 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:07.161059 4298 generic.go:356] "Generic (PLEG): container finished" podID="3317aadb-8f90-46e2-be55-fa9b81238900" containerID="a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0" exitCode=0 Nov 03 22:52:07.161203 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:07.161106 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" Nov 03 22:52:07.161203 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:07.161105 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" event={"ID":"3317aadb-8f90-46e2-be55-fa9b81238900","Type":"ContainerDied","Data":"a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0"} Nov 03 22:52:07.161277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:07.161210 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c" event={"ID":"3317aadb-8f90-46e2-be55-fa9b81238900","Type":"ContainerDied","Data":"006d6bcbc880e8b7a06cbccd74000aec22dd63ead037a7209fe681b0600123cd"} Nov 03 22:52:07.161277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:07.161238 4298 scope.go:122] "RemoveContainer" containerID="a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0" Nov 03 22:52:07.170820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:07.170792 4298 scope.go:122] "RemoveContainer" containerID="a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0" Nov 03 22:52:07.171096 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:07.171078 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0\": container with ID starting with a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0 not found: ID does not exist" containerID="a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0" Nov 03 22:52:07.171166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:07.171106 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0"} err="failed to get container status \"a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0\": rpc error: code = NotFound desc = could not find container \"a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0\": container with ID starting with a27bf5f4f362f7a755d349aa431f284147023ca02bf18024ec737568a216dee0 not found: ID does not exist" Nov 03 22:52:07.326910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:07.326869 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c"] Nov 03 22:52:07.330283 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:07.330261 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41bb2b-56677f99fd-x8s9c"] Nov 03 22:52:08.206613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:08.206576 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" podUID="1aedc98d-0cc8-492e-ad70-6b7b5a661b1f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:52:09.110410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.110373 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s"] Nov 03 22:52:09.110920 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.110903 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3317aadb-8f90-46e2-be55-fa9b81238900" containerName="payload-processing" Nov 03 22:52:09.110973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.110921 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3317aadb-8f90-46e2-be55-fa9b81238900" containerName="payload-processing" Nov 03 22:52:09.110973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.110936 4298 cpu_manager.go:414] "containerMap: removing container" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" Nov 03 22:52:09.110973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.110940 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" Nov 03 22:52:09.110973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.110948 4298 cpu_manager.go:414] "containerMap: removing container" podUID="bbbd19d3-a6c5-4036-8af3-c3cc75926b3b" containerName="revoke-keys" Nov 03 22:52:09.110973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.110952 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbd19d3-a6c5-4036-8af3-c3cc75926b3b" containerName="revoke-keys" Nov 03 22:52:09.110973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.110964 4298 cpu_manager.go:414] "containerMap: removing container" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" Nov 03 22:52:09.110973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.110969 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" Nov 03 22:52:09.111197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.110982 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c6639a62-b480-40e7-8c87-9aa9ba85160d" containerName="payload-processing" Nov 03 22:52:09.111197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.110987 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6639a62-b480-40e7-8c87-9aa9ba85160d" containerName="payload-processing" Nov 03 22:52:09.111197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.111060 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3317aadb-8f90-46e2-be55-fa9b81238900" containerName="payload-processing" Nov 03 22:52:09.111197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.111070 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" Nov 03 22:52:09.111197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.111079 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbd19d3-a6c5-4036-8af3-c3cc75926b3b" containerName="revoke-keys" Nov 03 22:52:09.111197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.111086 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe800041-c967-421b-a414-2a6aebb1d69c" containerName="istio-proxy" Nov 03 22:52:09.111197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.111092 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6639a62-b480-40e7-8c87-9aa9ba85160d" containerName="payload-processing" Nov 03 22:52:09.111197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.111099 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" Nov 03 22:52:09.115759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.115736 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:09.119535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.119498 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" containerID="cri-o://31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023" gracePeriod=45 Nov 03 22:52:09.125668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.125614 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrrwd\" (UniqueName: \"kubernetes.io/projected/5a9a9597-e202-4231-80f3-6c9cf7c915ca-kube-api-access-jrrwd\") pod \"maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:09.125783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.125721 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:09.125857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.125798 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:09.133841 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.133819 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3317aadb-8f90-46e2-be55-fa9b81238900" path="/var/lib/kubelet/pods/3317aadb-8f90-46e2-be55-fa9b81238900/volumes" Nov 03 22:52:09.173278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.173250 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s"] Nov 03 22:52:09.218456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.218424 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:52:09.218456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.218433 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:52:09.226306 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.226283 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:09.226441 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.226400 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrrwd\" (UniqueName: \"kubernetes.io/projected/5a9a9597-e202-4231-80f3-6c9cf7c915ca-kube-api-access-jrrwd\") pod \"maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:09.226505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.226455 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:09.227114 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.227090 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:09.228853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.228828 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:09.284376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.284334 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrrwd\" (UniqueName: \"kubernetes.io/projected/5a9a9597-e202-4231-80f3-6c9cf7c915ca-kube-api-access-jrrwd\") pod \"maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:09.383347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.383322 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:52:09.429295 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429271 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-model-cache\") pod \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " Nov 03 22:52:09.429449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429311 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tmp-dir\") pod \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " Nov 03 22:52:09.429449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429336 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-dshm\") pod \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " Nov 03 22:52:09.429449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429370 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-home\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-home\") pod \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " Nov 03 22:52:09.429449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429429 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-kube-api-access-zp4fm\" (UniqueName: \"kubernetes.io/projected/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-kube-api-access-zp4fm\") pod \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " Nov 03 22:52:09.429717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429462 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tls-certs\" (UniqueName: \"kubernetes.io/secret/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tls-certs\") pod \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\" (UID: \"293038f2-6fc8-4c35-a1fb-d9ac2b02553d\") " Nov 03 22:52:09.429717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429505 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:09.429717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429521 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-model-cache" pod "293038f2-6fc8-4c35-a1fb-d9ac2b02553d" (UID: "293038f2-6fc8-4c35-a1fb-d9ac2b02553d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.429717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429533 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tmp-dir" pod "293038f2-6fc8-4c35-a1fb-d9ac2b02553d" (UID: "293038f2-6fc8-4c35-a1fb-d9ac2b02553d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.429883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429736 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-home" pod "293038f2-6fc8-4c35-a1fb-d9ac2b02553d" (UID: "293038f2-6fc8-4c35-a1fb-d9ac2b02553d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.429917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429894 4298 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tmp-dir\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.429917 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429909 4298 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-home\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.429987 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.429917 4298 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-model-cache\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.431551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.431526 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-dshm" pod "293038f2-6fc8-4c35-a1fb-d9ac2b02553d" (UID: "293038f2-6fc8-4c35-a1fb-d9ac2b02553d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.431694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.431588 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tls-certs" pod "293038f2-6fc8-4c35-a1fb-d9ac2b02553d" (UID: "293038f2-6fc8-4c35-a1fb-d9ac2b02553d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:09.431694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.431592 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-kube-api-access-zp4fm" pod "293038f2-6fc8-4c35-a1fb-d9ac2b02553d" (UID: "293038f2-6fc8-4c35-a1fb-d9ac2b02553d"). InnerVolumeSpecName "kube-api-access-zp4fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:09.531473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.531440 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4fm\" (UniqueName: \"kubernetes.io/projected/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-kube-api-access-zp4fm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.531473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.531476 4298 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-tls-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.531690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.531491 4298 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/293038f2-6fc8-4c35-a1fb-d9ac2b02553d-dshm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.691370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.691305 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:09.796719 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:52:09.796686 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9a9597_e202_4231_80f3_6c9cf7c915ca.slice/crio-3a9b36d11957db083b63d9ed9212d6b079701fd21b6b22b20fc74332cd2a39da WatchSource:0}: Error finding container 3a9b36d11957db083b63d9ed9212d6b079701fd21b6b22b20fc74332cd2a39da: Status 404 returned error can't find the container with id 3a9b36d11957db083b63d9ed9212d6b079701fd21b6b22b20fc74332cd2a39da Nov 03 22:52:09.800050 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:09.800026 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s"] Nov 03 22:52:10.190131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.190092 4298 generic.go:356] "Generic (PLEG): container finished" podID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerID="31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023" exitCode=0 Nov 03 22:52:10.190327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.190143 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" Nov 03 22:52:10.190327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.190185 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" event={"ID":"293038f2-6fc8-4c35-a1fb-d9ac2b02553d","Type":"ContainerDied","Data":"31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023"} Nov 03 22:52:10.190327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.190236 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx" event={"ID":"293038f2-6fc8-4c35-a1fb-d9ac2b02553d","Type":"ContainerDied","Data":"24d86deedce6bff8fa60d451149bf192305f9361955b08918508115b4300f5c8"} Nov 03 22:52:10.190327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.190259 4298 scope.go:122] "RemoveContainer" containerID="31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023" Nov 03 22:52:10.192117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.192087 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" event={"ID":"5a9a9597-e202-4231-80f3-6c9cf7c915ca","Type":"ContainerStarted","Data":"1ab7df3e72607640a48f960c94780f8ee7b629dbb64bd3c0c559464d96654b05"} Nov 03 22:52:10.192247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.192132 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" event={"ID":"5a9a9597-e202-4231-80f3-6c9cf7c915ca","Type":"ContainerStarted","Data":"3a9b36d11957db083b63d9ed9212d6b079701fd21b6b22b20fc74332cd2a39da"} Nov 03 22:52:10.201249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.201232 4298 scope.go:122] "RemoveContainer" containerID="31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023" Nov 03 22:52:10.201541 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:10.201524 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023\": container with ID starting with 31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023 not found: ID does not exist" containerID="31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023" Nov 03 22:52:10.201587 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.201549 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023"} err="failed to get container status \"31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023\": rpc error: code = NotFound desc = could not find container \"31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023\": container with ID starting with 31c8bb802821fde738650057254c6f7699bdbd1a4740aa81c806e011c5f84023 not found: ID does not exist" Nov 03 22:52:10.359551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.359454 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" podStartSLOduration=1.359440644 podStartE2EDuration="1.359440644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:10.355777038 +0000 UTC m=+2687.336510690" watchObservedRunningTime="2026-11-03 22:52:10.359440644 +0000 UTC m=+2687.340174297" Nov 03 22:52:10.471470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.471436 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx"] Nov 03 22:52:10.476856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:10.476824 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/simulator-w3-41bb2b-kserve-68ff6ddbcf-2smqx"] Nov 03 22:52:11.136215 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:11.136124 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" path="/var/lib/kubelet/pods/293038f2-6fc8-4c35-a1fb-d9ac2b02553d/volumes" Nov 03 22:52:11.202222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:11.202188 4298 generic.go:356] "Generic (PLEG): container finished" podID="5a9a9597-e202-4231-80f3-6c9cf7c915ca" containerID="1ab7df3e72607640a48f960c94780f8ee7b629dbb64bd3c0c559464d96654b05" exitCode=0 Nov 03 22:52:11.202442 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:11.202278 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" event={"ID":"5a9a9597-e202-4231-80f3-6c9cf7c915ca","Type":"ContainerDied","Data":"1ab7df3e72607640a48f960c94780f8ee7b629dbb64bd3c0c559464d96654b05"} Nov 03 22:52:11.211839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:11.211796 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:12.355599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.355573 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:12.468394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.468308 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-openshift-ca\") pod \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " Nov 03 22:52:12.468394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.468387 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-xks-ca\") pod \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " Nov 03 22:52:12.468638 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.468435 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9a9597-e202-4231-80f3-6c9cf7c915ca-kube-api-access-jrrwd\" (UniqueName: \"kubernetes.io/projected/5a9a9597-e202-4231-80f3-6c9cf7c915ca-kube-api-access-jrrwd\") pod \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\" (UID: \"5a9a9597-e202-4231-80f3-6c9cf7c915ca\") " Nov 03 22:52:12.468722 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.468639 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-openshift-ca" pod "5a9a9597-e202-4231-80f3-6c9cf7c915ca" (UID: "5a9a9597-e202-4231-80f3-6c9cf7c915ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:12.469282 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.469259 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.470799 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.470766 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9a9597-e202-4231-80f3-6c9cf7c915ca-kube-api-access-jrrwd" pod "5a9a9597-e202-4231-80f3-6c9cf7c915ca" (UID: "5a9a9597-e202-4231-80f3-6c9cf7c915ca"). InnerVolumeSpecName "kube-api-access-jrrwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:12.470910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.470856 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-xks-ca" pod "5a9a9597-e202-4231-80f3-6c9cf7c915ca" (UID: "5a9a9597-e202-4231-80f3-6c9cf7c915ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:12.484613 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.484585 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst"] Nov 03 22:52:12.485287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.485264 4298 cpu_manager.go:414] "containerMap: removing container" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" Nov 03 22:52:12.485287 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.485288 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36642f0e-de5f-4793-9398-0965f2d8ea70" containerName="istio-proxy" Nov 03 22:52:12.485474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.485313 4298 cpu_manager.go:414] "containerMap: removing container" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" Nov 03 22:52:12.485474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.485321 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" Nov 03 22:52:12.485474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.485361 4298 cpu_manager.go:414] "containerMap: removing container" podUID="5a9a9597-e202-4231-80f3-6c9cf7c915ca" containerName="revoke-keys" Nov 03 22:52:12.485474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.485369 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9a9597-e202-4231-80f3-6c9cf7c915ca" containerName="revoke-keys" Nov 03 22:52:12.485782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.485479 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9a9597-e202-4231-80f3-6c9cf7c915ca" containerName="revoke-keys" Nov 03 22:52:12.485782 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.485498 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="293038f2-6fc8-4c35-a1fb-d9ac2b02553d" containerName="main" Nov 03 22:52:12.489826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.489803 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:12.566334 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.566293 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst"] Nov 03 22:52:12.570392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.570360 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc0b00f1-vnfst\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:12.570673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.570463 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc0b00f1-vnfst\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:12.570673 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.570603 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v548n\" (UniqueName: \"kubernetes.io/projected/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-kube-api-access-v548n\") pod \"maas-api-revoke-sub-fc0b00f1-vnfst\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:12.570816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.570716 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9a9597-e202-4231-80f3-6c9cf7c915ca-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.570816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.570737 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrrwd\" (UniqueName: \"kubernetes.io/projected/5a9a9597-e202-4231-80f3-6c9cf7c915ca-kube-api-access-jrrwd\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.572893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.572855 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb"] Nov 03 22:52:12.577999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.577967 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:12.627958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.627927 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb"] Nov 03 22:52:12.672079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.672049 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v548n\" (UniqueName: \"kubernetes.io/projected/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-kube-api-access-v548n\") pod \"maas-api-revoke-sub-fc0b00f1-vnfst\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:12.672301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.672110 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc0b00f1-vnfst\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:12.672301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.672175 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d9df21-fdqqb\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:12.672301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.672206 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlgt\" (UniqueName: \"kubernetes.io/projected/58871bb6-bc1c-42ad-9432-9bf28802fa2a-kube-api-access-znlgt\") pod \"maas-api-revoke-sub-88d9df21-fdqqb\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:12.672451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.672427 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc0b00f1-vnfst\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:12.672527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.672509 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d9df21-fdqqb\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:12.672851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.672830 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc0b00f1-vnfst\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:12.674826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.674807 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc0b00f1-vnfst\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:12.730776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.730709 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v548n\" (UniqueName: \"kubernetes.io/projected/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-kube-api-access-v548n\") pod \"maas-api-revoke-sub-fc0b00f1-vnfst\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:12.774276 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.774240 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d9df21-fdqqb\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:12.774537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.774286 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlgt\" (UniqueName: \"kubernetes.io/projected/58871bb6-bc1c-42ad-9432-9bf28802fa2a-kube-api-access-znlgt\") pod \"maas-api-revoke-sub-88d9df21-fdqqb\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:12.774537 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.774324 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d9df21-fdqqb\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:12.774956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.774935 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d9df21-fdqqb\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:12.776608 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.776584 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d9df21-fdqqb\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:12.832364 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.832332 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlgt\" (UniqueName: \"kubernetes.io/projected/58871bb6-bc1c-42ad-9432-9bf28802fa2a-kube-api-access-znlgt\") pod \"maas-api-revoke-sub-88d9df21-fdqqb\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:12.844179 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.844159 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:12.874380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.874356 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:12.893860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:12.893833 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:13.212924 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:52:13.211877 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a19f04_01d5_4af7_aff6_22f1d4bc3d20.slice/crio-08014a06412e3d91355e2583c3dfca46c6f984a1d6908aba1c5355df06072c6b WatchSource:0}: Error finding container 08014a06412e3d91355e2583c3dfca46c6f984a1d6908aba1c5355df06072c6b: Status 404 returned error can't find the container with id 08014a06412e3d91355e2583c3dfca46c6f984a1d6908aba1c5355df06072c6b Nov 03 22:52:13.216592 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.216413 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst"] Nov 03 22:52:13.217401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.217379 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" event={"ID":"5a9a9597-e202-4231-80f3-6c9cf7c915ca","Type":"ContainerDied","Data":"3a9b36d11957db083b63d9ed9212d6b079701fd21b6b22b20fc74332cd2a39da"} Nov 03 22:52:13.217484 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.217396 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s" Nov 03 22:52:13.217484 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.217422 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a9b36d11957db083b63d9ed9212d6b079701fd21b6b22b20fc74332cd2a39da" Nov 03 22:52:13.253755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.253723 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:13.307332 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:52:13.307303 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58871bb6_bc1c_42ad_9432_9bf28802fa2a.slice/crio-f251f22092adac21de3a0b139205c67a9dfd8d2bbceeffa81f90ae98a5218fa3 WatchSource:0}: Error finding container f251f22092adac21de3a0b139205c67a9dfd8d2bbceeffa81f90ae98a5218fa3: Status 404 returned error can't find the container with id f251f22092adac21de3a0b139205c67a9dfd8d2bbceeffa81f90ae98a5218fa3 Nov 03 22:52:13.311543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.311473 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb"] Nov 03 22:52:13.507657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.507612 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp"] Nov 03 22:52:13.535989 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.535957 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b30a14-xbprp"] Nov 03 22:52:13.549216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.549195 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj"] Nov 03 22:52:13.622792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.622762 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:13.646712 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.646686 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz"] Nov 03 22:52:13.812248 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.812205 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:13.994479 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:13.994447 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:14.179914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:14.179886 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:14.225691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:14.225636 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" event={"ID":"58871bb6-bc1c-42ad-9432-9bf28802fa2a","Type":"ContainerStarted","Data":"8035f14d0bab5038e96a4b4b5590b0dcf788b3ecf354fcbfa8a606f403727eea"} Nov 03 22:52:14.225887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:14.225721 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" event={"ID":"58871bb6-bc1c-42ad-9432-9bf28802fa2a","Type":"ContainerStarted","Data":"f251f22092adac21de3a0b139205c67a9dfd8d2bbceeffa81f90ae98a5218fa3"} Nov 03 22:52:14.227440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:14.227412 4298 generic.go:356] "Generic (PLEG): container finished" podID="33a19f04-01d5-4af7-aff6-22f1d4bc3d20" containerID="397d4faae2d895a2a8d787aa304107f869ea0a272bd9b03763f12df21c525c47" exitCode=0 Nov 03 22:52:14.227621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:14.227453 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" event={"ID":"33a19f04-01d5-4af7-aff6-22f1d4bc3d20","Type":"ContainerDied","Data":"397d4faae2d895a2a8d787aa304107f869ea0a272bd9b03763f12df21c525c47"} Nov 03 22:52:14.227621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:14.227505 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" event={"ID":"33a19f04-01d5-4af7-aff6-22f1d4bc3d20","Type":"ContainerStarted","Data":"08014a06412e3d91355e2583c3dfca46c6f984a1d6908aba1c5355df06072c6b"} Nov 03 22:52:14.343085 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:14.343041 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" podStartSLOduration=2.343026289 podStartE2EDuration="2.343026289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:14.340174585 +0000 UTC m=+2691.320908238" watchObservedRunningTime="2026-11-03 22:52:14.343026289 +0000 UTC m=+2691.323759941" Nov 03 22:52:15.134748 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.134708 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39aaecd-14df-4068-b54a-bec762e4a528" path="/var/lib/kubelet/pods/b39aaecd-14df-4068-b54a-bec762e4a528/volumes" Nov 03 22:52:15.235870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.235831 4298 generic.go:356] "Generic (PLEG): container finished" podID="58871bb6-bc1c-42ad-9432-9bf28802fa2a" containerID="8035f14d0bab5038e96a4b4b5590b0dcf788b3ecf354fcbfa8a606f403727eea" exitCode=0 Nov 03 22:52:15.236042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.235877 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" event={"ID":"58871bb6-bc1c-42ad-9432-9bf28802fa2a","Type":"ContainerDied","Data":"8035f14d0bab5038e96a4b4b5590b0dcf788b3ecf354fcbfa8a606f403727eea"} Nov 03 22:52:15.393034 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.392971 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:15.516750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.516722 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-openshift-ca\") pod \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " Nov 03 22:52:15.516960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.516830 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-kube-api-access-v548n\" (UniqueName: \"kubernetes.io/projected/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-kube-api-access-v548n\") pod \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " Nov 03 22:52:15.516960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.516903 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-xks-ca\") pod \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\" (UID: \"33a19f04-01d5-4af7-aff6-22f1d4bc3d20\") " Nov 03 22:52:15.517161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.517138 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-openshift-ca" pod "33a19f04-01d5-4af7-aff6-22f1d4bc3d20" (UID: "33a19f04-01d5-4af7-aff6-22f1d4bc3d20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:15.517366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.517343 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:15.520535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.520509 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-xks-ca" pod "33a19f04-01d5-4af7-aff6-22f1d4bc3d20" (UID: "33a19f04-01d5-4af7-aff6-22f1d4bc3d20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:15.520794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.520773 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-kube-api-access-v548n" pod "33a19f04-01d5-4af7-aff6-22f1d4bc3d20" (UID: "33a19f04-01d5-4af7-aff6-22f1d4bc3d20"). InnerVolumeSpecName "kube-api-access-v548n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:15.618459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.618413 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v548n\" (UniqueName: \"kubernetes.io/projected/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-kube-api-access-v548n\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:15.618459 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:15.618456 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a19f04-01d5-4af7-aff6-22f1d4bc3d20-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:16.245079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.245041 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" event={"ID":"33a19f04-01d5-4af7-aff6-22f1d4bc3d20","Type":"ContainerDied","Data":"08014a06412e3d91355e2583c3dfca46c6f984a1d6908aba1c5355df06072c6b"} Nov 03 22:52:16.245079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.245066 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst" Nov 03 22:52:16.245543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.245095 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08014a06412e3d91355e2583c3dfca46c6f984a1d6908aba1c5355df06072c6b" Nov 03 22:52:16.385547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.385519 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:16.528798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.528052 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58871bb6-bc1c-42ad-9432-9bf28802fa2a-kube-api-access-znlgt\" (UniqueName: \"kubernetes.io/projected/58871bb6-bc1c-42ad-9432-9bf28802fa2a-kube-api-access-znlgt\") pod \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " Nov 03 22:52:16.528798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.528152 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-xks-ca\") pod \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " Nov 03 22:52:16.528798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.528241 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-openshift-ca\") pod \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\" (UID: \"58871bb6-bc1c-42ad-9432-9bf28802fa2a\") " Nov 03 22:52:16.528798 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.528671 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-openshift-ca" pod "58871bb6-bc1c-42ad-9432-9bf28802fa2a" (UID: "58871bb6-bc1c-42ad-9432-9bf28802fa2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:16.530856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.530823 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58871bb6-bc1c-42ad-9432-9bf28802fa2a-kube-api-access-znlgt" pod "58871bb6-bc1c-42ad-9432-9bf28802fa2a" (UID: "58871bb6-bc1c-42ad-9432-9bf28802fa2a"). InnerVolumeSpecName "kube-api-access-znlgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:16.531028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.530900 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-xks-ca" pod "58871bb6-bc1c-42ad-9432-9bf28802fa2a" (UID: "58871bb6-bc1c-42ad-9432-9bf28802fa2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:16.629753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.629722 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:16.629753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.629751 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znlgt\" (UniqueName: \"kubernetes.io/projected/58871bb6-bc1c-42ad-9432-9bf28802fa2a-kube-api-access-znlgt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:16.629753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:16.629762 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58871bb6-bc1c-42ad-9432-9bf28802fa2a-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:17.252999 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:17.252968 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" Nov 03 22:52:17.253502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:17.253004 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb" event={"ID":"58871bb6-bc1c-42ad-9432-9bf28802fa2a","Type":"ContainerDied","Data":"f251f22092adac21de3a0b139205c67a9dfd8d2bbceeffa81f90ae98a5218fa3"} Nov 03 22:52:17.253502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:17.253061 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f251f22092adac21de3a0b139205c67a9dfd8d2bbceeffa81f90ae98a5218fa3" Nov 03 22:52:17.551334 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:17.551299 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:52:18.206080 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:18.206046 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" podUID="1aedc98d-0cc8-492e-ad70-6b7b5a661b1f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:52:18.550181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:18.550081 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" podUID="82cb7348-5057-4da0-9c8d-fb00195e3ccc" containerName="payload-processing" containerID="cri-o://9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543" gracePeriod=25 Nov 03 22:52:18.621077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:18.621047 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:18.647926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:18.647887 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" podUID="eea684a8-7762-45a7-8a60-a41d2e76cd4b" containerName="payload-pre-processing" containerID="cri-o://36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89" gracePeriod=25 Nov 03 22:52:18.805428 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:18.805360 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:52:18.905292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:18.905266 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:52:18.958049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:18.958014 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82cb7348-5057-4da0-9c8d-fb00195e3ccc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cb7348-5057-4da0-9c8d-fb00195e3ccc-plugins-config-volume\") pod \"82cb7348-5057-4da0-9c8d-fb00195e3ccc\" (UID: \"82cb7348-5057-4da0-9c8d-fb00195e3ccc\") " Nov 03 22:52:18.958235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:18.958075 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82cb7348-5057-4da0-9c8d-fb00195e3ccc-kube-api-access-vfhtd\" (UniqueName: \"kubernetes.io/projected/82cb7348-5057-4da0-9c8d-fb00195e3ccc-kube-api-access-vfhtd\") pod \"82cb7348-5057-4da0-9c8d-fb00195e3ccc\" (UID: \"82cb7348-5057-4da0-9c8d-fb00195e3ccc\") " Nov 03 22:52:18.958504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:18.958479 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82cb7348-5057-4da0-9c8d-fb00195e3ccc-plugins-config-volume" pod "82cb7348-5057-4da0-9c8d-fb00195e3ccc" (UID: "82cb7348-5057-4da0-9c8d-fb00195e3ccc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:18.960162 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:18.960135 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cb7348-5057-4da0-9c8d-fb00195e3ccc-kube-api-access-vfhtd" pod "82cb7348-5057-4da0-9c8d-fb00195e3ccc" (UID: "82cb7348-5057-4da0-9c8d-fb00195e3ccc"). InnerVolumeSpecName "kube-api-access-vfhtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:18.963103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:18.963081 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:19.059526 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.059446 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea684a8-7762-45a7-8a60-a41d2e76cd4b-kube-api-access-mvjgm\" (UniqueName: \"kubernetes.io/projected/eea684a8-7762-45a7-8a60-a41d2e76cd4b-kube-api-access-mvjgm\") pod \"eea684a8-7762-45a7-8a60-a41d2e76cd4b\" (UID: \"eea684a8-7762-45a7-8a60-a41d2e76cd4b\") " Nov 03 22:52:19.059526 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.059512 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea684a8-7762-45a7-8a60-a41d2e76cd4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea684a8-7762-45a7-8a60-a41d2e76cd4b-plugins-config-volume\") pod \"eea684a8-7762-45a7-8a60-a41d2e76cd4b\" (UID: \"eea684a8-7762-45a7-8a60-a41d2e76cd4b\") " Nov 03 22:52:19.059857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.059839 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cb7348-5057-4da0-9c8d-fb00195e3ccc-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:19.059857 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.059858 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfhtd\" (UniqueName: \"kubernetes.io/projected/82cb7348-5057-4da0-9c8d-fb00195e3ccc-kube-api-access-vfhtd\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:19.060029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.060001 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea684a8-7762-45a7-8a60-a41d2e76cd4b-plugins-config-volume" pod "eea684a8-7762-45a7-8a60-a41d2e76cd4b" (UID: "eea684a8-7762-45a7-8a60-a41d2e76cd4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:19.061557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.061534 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea684a8-7762-45a7-8a60-a41d2e76cd4b-kube-api-access-mvjgm" pod "eea684a8-7762-45a7-8a60-a41d2e76cd4b" (UID: "eea684a8-7762-45a7-8a60-a41d2e76cd4b"). InnerVolumeSpecName "kube-api-access-mvjgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:19.160941 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.160914 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvjgm\" (UniqueName: \"kubernetes.io/projected/eea684a8-7762-45a7-8a60-a41d2e76cd4b-kube-api-access-mvjgm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:19.160941 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.160939 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea684a8-7762-45a7-8a60-a41d2e76cd4b-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:19.266827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.266791 4298 generic.go:356] "Generic (PLEG): container finished" podID="eea684a8-7762-45a7-8a60-a41d2e76cd4b" containerID="36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89" exitCode=0 Nov 03 22:52:19.267020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.266843 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" Nov 03 22:52:19.267020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.266876 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" event={"ID":"eea684a8-7762-45a7-8a60-a41d2e76cd4b","Type":"ContainerDied","Data":"36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89"} Nov 03 22:52:19.267020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.266940 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz" event={"ID":"eea684a8-7762-45a7-8a60-a41d2e76cd4b","Type":"ContainerDied","Data":"d25ee62c4be26df7601a7ec815be3f88ccf7c517ae33e01e8df5f37424b062cc"} Nov 03 22:52:19.267020 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.266972 4298 scope.go:122] "RemoveContainer" containerID="36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89" Nov 03 22:52:19.268309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.268289 4298 generic.go:356] "Generic (PLEG): container finished" podID="82cb7348-5057-4da0-9c8d-fb00195e3ccc" containerID="9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543" exitCode=0 Nov 03 22:52:19.268398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.268345 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" Nov 03 22:52:19.268398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.268365 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" event={"ID":"82cb7348-5057-4da0-9c8d-fb00195e3ccc","Type":"ContainerDied","Data":"9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543"} Nov 03 22:52:19.268482 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.268418 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj" event={"ID":"82cb7348-5057-4da0-9c8d-fb00195e3ccc","Type":"ContainerDied","Data":"723d3938ca92f367b5ffedecba5dd13050e12d0e3941d00f41d44ac5a0bda3ca"} Nov 03 22:52:19.277350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.277328 4298 scope.go:122] "RemoveContainer" containerID="36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89" Nov 03 22:52:19.277622 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:19.277603 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89\": container with ID starting with 36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89 not found: ID does not exist" containerID="36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89" Nov 03 22:52:19.277704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.277630 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89"} err="failed to get container status \"36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89\": rpc error: code = NotFound desc = could not find container \"36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89\": container with ID starting with 36c3a28fc70a1992becbbd6c063df331ce58c2f9aa97c1cf316a794d2018df89 not found: ID does not exist" Nov 03 22:52:19.277704 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.277667 4298 scope.go:122] "RemoveContainer" containerID="9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543" Nov 03 22:52:19.286277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.286259 4298 scope.go:122] "RemoveContainer" containerID="9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543" Nov 03 22:52:19.286552 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:19.286523 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543\": container with ID starting with 9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543 not found: ID does not exist" containerID="9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543" Nov 03 22:52:19.286603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.286551 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543"} err="failed to get container status \"9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543\": rpc error: code = NotFound desc = could not find container \"9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543\": container with ID starting with 9c35066523f11ed85fd624412d9eae2eeac802ae7afd016f9f46066aa6aa6543 not found: ID does not exist" Nov 03 22:52:19.433492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.433396 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj"] Nov 03 22:52:19.483442 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.483410 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-fac3b6-558cf8d644-cgrvj"] Nov 03 22:52:19.593025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.592997 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz"] Nov 03 22:52:19.596314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.596271 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-fac3b6-86dc68899f-5fkrz"] Nov 03 22:52:19.710481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.710446 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:19.892266 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.892236 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:19.972037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.971945 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr"] Nov 03 22:52:19.972228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:19.972204 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" containerID="cri-o://217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840" gracePeriod=30 Nov 03 22:52:20.094664 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:20.085932 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:20.560733 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:20.560702 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:20.869042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:20.868964 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d"] Nov 03 22:52:21.005444 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.005413 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7"] Nov 03 22:52:21.005904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.005890 4298 cpu_manager.go:414] "containerMap: removing container" podUID="eea684a8-7762-45a7-8a60-a41d2e76cd4b" containerName="payload-pre-processing" Nov 03 22:52:21.005947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.005905 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea684a8-7762-45a7-8a60-a41d2e76cd4b" containerName="payload-pre-processing" Nov 03 22:52:21.005947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.005916 4298 cpu_manager.go:414] "containerMap: removing container" podUID="33a19f04-01d5-4af7-aff6-22f1d4bc3d20" containerName="revoke-keys" Nov 03 22:52:21.005947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.005921 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a19f04-01d5-4af7-aff6-22f1d4bc3d20" containerName="revoke-keys" Nov 03 22:52:21.005947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.005934 4298 cpu_manager.go:414] "containerMap: removing container" podUID="82cb7348-5057-4da0-9c8d-fb00195e3ccc" containerName="payload-processing" Nov 03 22:52:21.005947 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.005939 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82cb7348-5057-4da0-9c8d-fb00195e3ccc" containerName="payload-processing" Nov 03 22:52:21.006133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.005952 4298 cpu_manager.go:414] "containerMap: removing container" podUID="58871bb6-bc1c-42ad-9432-9bf28802fa2a" containerName="revoke-keys" Nov 03 22:52:21.006133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.005957 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58871bb6-bc1c-42ad-9432-9bf28802fa2a" containerName="revoke-keys" Nov 03 22:52:21.006133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.006025 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea684a8-7762-45a7-8a60-a41d2e76cd4b" containerName="payload-pre-processing" Nov 03 22:52:21.006133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.006035 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a19f04-01d5-4af7-aff6-22f1d4bc3d20" containerName="revoke-keys" Nov 03 22:52:21.006133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.006043 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="82cb7348-5057-4da0-9c8d-fb00195e3ccc" containerName="payload-processing" Nov 03 22:52:21.006133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.006050 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="58871bb6-bc1c-42ad-9432-9bf28802fa2a" containerName="revoke-keys" Nov 03 22:52:21.009382 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.009357 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:21.062533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.062506 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:52:21.062720 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.062638 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:52:21.068339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.068198 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7"] Nov 03 22:52:21.083929 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.083903 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:21.084077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.083961 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvg2l\" (UniqueName: \"kubernetes.io/projected/af323b31-3c27-4a31-87e8-94a5ae96aafc-kube-api-access-vvg2l\") pod \"maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:21.084145 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.084122 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:21.134076 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.134007 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82cb7348-5057-4da0-9c8d-fb00195e3ccc" path="/var/lib/kubelet/pods/82cb7348-5057-4da0-9c8d-fb00195e3ccc/volumes" Nov 03 22:52:21.134448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.134432 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea684a8-7762-45a7-8a60-a41d2e76cd4b" path="/var/lib/kubelet/pods/eea684a8-7762-45a7-8a60-a41d2e76cd4b/volumes" Nov 03 22:52:21.167357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.167333 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:21.187611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.185388 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:21.199243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.187745 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvg2l\" (UniqueName: \"kubernetes.io/projected/af323b31-3c27-4a31-87e8-94a5ae96aafc-kube-api-access-vvg2l\") pod \"maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:21.199243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.188490 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:21.199243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.189073 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:21.199243 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.189309 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:21.244637 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.244604 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvg2l\" (UniqueName: \"kubernetes.io/projected/af323b31-3c27-4a31-87e8-94a5ae96aafc-kube-api-access-vvg2l\") pod \"maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:21.326489 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.326460 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:21.345818 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.345786 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:21.694362 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:52:21.694321 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf323b31_3c27_4a31_87e8_94a5ae96aafc.slice/crio-2f3a54bf062ddd3c617389e078c98ebfa226559ea54ba8d1f83a5de93c9ae1a7 WatchSource:0}: Error finding container 2f3a54bf062ddd3c617389e078c98ebfa226559ea54ba8d1f83a5de93c9ae1a7: Status 404 returned error can't find the container with id 2f3a54bf062ddd3c617389e078c98ebfa226559ea54ba8d1f83a5de93c9ae1a7 Nov 03 22:52:21.699050 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:21.699029 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7"] Nov 03 22:52:22.293755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:22.293624 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" event={"ID":"af323b31-3c27-4a31-87e8-94a5ae96aafc","Type":"ContainerStarted","Data":"1d6dad44df1aa83aae24a97e4ff0542e871a9b89f1214c2552bba16fac8bf450"} Nov 03 22:52:22.293755 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:22.293718 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" event={"ID":"af323b31-3c27-4a31-87e8-94a5ae96aafc","Type":"ContainerStarted","Data":"2f3a54bf062ddd3c617389e078c98ebfa226559ea54ba8d1f83a5de93c9ae1a7"} Nov 03 22:52:22.409082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:22.409028 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" podStartSLOduration=2.409012608 podStartE2EDuration="2.409012608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:22.406074064 +0000 UTC m=+2699.386807716" watchObservedRunningTime="2026-11-03 22:52:22.409012608 +0000 UTC m=+2699.389746261" Nov 03 22:52:23.242165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:23.242132 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:23.302378 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:23.302343 4298 generic.go:356] "Generic (PLEG): container finished" podID="af323b31-3c27-4a31-87e8-94a5ae96aafc" containerID="1d6dad44df1aa83aae24a97e4ff0542e871a9b89f1214c2552bba16fac8bf450" exitCode=0 Nov 03 22:52:23.302795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:23.302427 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" event={"ID":"af323b31-3c27-4a31-87e8-94a5ae96aafc","Type":"ContainerDied","Data":"1d6dad44df1aa83aae24a97e4ff0542e871a9b89f1214c2552bba16fac8bf450"} Nov 03 22:52:24.211314 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:24.211283 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:24.446558 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:24.446534 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:24.523428 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:24.523339 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af323b31-3c27-4a31-87e8-94a5ae96aafc-kube-api-access-vvg2l\" (UniqueName: \"kubernetes.io/projected/af323b31-3c27-4a31-87e8-94a5ae96aafc-kube-api-access-vvg2l\") pod \"af323b31-3c27-4a31-87e8-94a5ae96aafc\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " Nov 03 22:52:24.523594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:24.523458 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-openshift-ca\") pod \"af323b31-3c27-4a31-87e8-94a5ae96aafc\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " Nov 03 22:52:24.523594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:24.523517 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-xks-ca\") pod \"af323b31-3c27-4a31-87e8-94a5ae96aafc\" (UID: \"af323b31-3c27-4a31-87e8-94a5ae96aafc\") " Nov 03 22:52:24.523826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:24.523790 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-openshift-ca" pod "af323b31-3c27-4a31-87e8-94a5ae96aafc" (UID: "af323b31-3c27-4a31-87e8-94a5ae96aafc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:24.524002 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:24.523981 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:24.525577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:24.525555 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af323b31-3c27-4a31-87e8-94a5ae96aafc-kube-api-access-vvg2l" pod "af323b31-3c27-4a31-87e8-94a5ae96aafc" (UID: "af323b31-3c27-4a31-87e8-94a5ae96aafc"). InnerVolumeSpecName "kube-api-access-vvg2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:24.525632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:24.525615 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-xks-ca" pod "af323b31-3c27-4a31-87e8-94a5ae96aafc" (UID: "af323b31-3c27-4a31-87e8-94a5ae96aafc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:24.625393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:24.625358 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvg2l\" (UniqueName: \"kubernetes.io/projected/af323b31-3c27-4a31-87e8-94a5ae96aafc-kube-api-access-vvg2l\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:24.625393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:24.625390 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af323b31-3c27-4a31-87e8-94a5ae96aafc-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:25.231907 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.231881 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:52:25.317073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.316997 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" Nov 03 22:52:25.317073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.317013 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7" event={"ID":"af323b31-3c27-4a31-87e8-94a5ae96aafc","Type":"ContainerDied","Data":"2f3a54bf062ddd3c617389e078c98ebfa226559ea54ba8d1f83a5de93c9ae1a7"} Nov 03 22:52:25.317073 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.317065 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f3a54bf062ddd3c617389e078c98ebfa226559ea54ba8d1f83a5de93c9ae1a7" Nov 03 22:52:25.319008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.318958 4298 generic.go:356] "Generic (PLEG): container finished" podID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerID="217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840" exitCode=0 Nov 03 22:52:25.319008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.318992 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" event={"ID":"5659da87-ec7e-4434-a21c-c33ac6aacd20","Type":"ContainerDied","Data":"217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840"} Nov 03 22:52:25.319008 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.319005 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" Nov 03 22:52:25.319300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.319056 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr" event={"ID":"5659da87-ec7e-4434-a21c-c33ac6aacd20","Type":"ContainerDied","Data":"3c3cd91e98105d5ebb08fb34cb0de9e9b563ce042b8b7d96f808c330ff566e84"} Nov 03 22:52:25.319300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.319089 4298 scope.go:122] "RemoveContainer" containerID="217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840" Nov 03 22:52:25.328824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.328804 4298 scope.go:122] "RemoveContainer" containerID="cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55" Nov 03 22:52:25.332180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332159 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-certs\") pod \"5659da87-ec7e-4434-a21c-c33ac6aacd20\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " Nov 03 22:52:25.332302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332198 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5659da87-ec7e-4434-a21c-c33ac6aacd20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5659da87-ec7e-4434-a21c-c33ac6aacd20-istiod-ca-cert\") pod \"5659da87-ec7e-4434-a21c-c33ac6aacd20\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " Nov 03 22:52:25.332302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332228 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-socket\") pod \"5659da87-ec7e-4434-a21c-c33ac6aacd20\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " Nov 03 22:52:25.332302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332265 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-data\") pod \"5659da87-ec7e-4434-a21c-c33ac6aacd20\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " Nov 03 22:52:25.332481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332298 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-kube-api-access-9tnxt\" (UniqueName: \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-kube-api-access-9tnxt\") pod \"5659da87-ec7e-4434-a21c-c33ac6aacd20\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " Nov 03 22:52:25.332481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332389 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-envoy\") pod \"5659da87-ec7e-4434-a21c-c33ac6aacd20\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " Nov 03 22:52:25.332481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332434 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-certs" pod "5659da87-ec7e-4434-a21c-c33ac6aacd20" (UID: "5659da87-ec7e-4434-a21c-c33ac6aacd20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:25.332481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332472 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-token\" (UniqueName: \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-token\") pod \"5659da87-ec7e-4434-a21c-c33ac6aacd20\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " Nov 03 22:52:25.332714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332511 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-socket" pod "5659da87-ec7e-4434-a21c-c33ac6aacd20" (UID: "5659da87-ec7e-4434-a21c-c33ac6aacd20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:25.332714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332523 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-data" pod "5659da87-ec7e-4434-a21c-c33ac6aacd20" (UID: "5659da87-ec7e-4434-a21c-c33ac6aacd20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:25.332714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332541 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-credential-socket\") pod \"5659da87-ec7e-4434-a21c-c33ac6aacd20\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " Nov 03 22:52:25.332714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332583 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-podinfo\") pod \"5659da87-ec7e-4434-a21c-c33ac6aacd20\" (UID: \"5659da87-ec7e-4434-a21c-c33ac6aacd20\") " Nov 03 22:52:25.332714 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332628 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5659da87-ec7e-4434-a21c-c33ac6aacd20-istiod-ca-cert" pod "5659da87-ec7e-4434-a21c-c33ac6aacd20" (UID: "5659da87-ec7e-4434-a21c-c33ac6aacd20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:25.332967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.332913 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-credential-socket" pod "5659da87-ec7e-4434-a21c-c33ac6aacd20" (UID: "5659da87-ec7e-4434-a21c-c33ac6aacd20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:25.333037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.333019 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:25.333084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.333042 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:25.333084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.333056 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5659da87-ec7e-4434-a21c-c33ac6aacd20-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:25.333084 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.333069 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:25.333238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.333083 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:25.334610 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.334588 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-kube-api-access-9tnxt" pod "5659da87-ec7e-4434-a21c-c33ac6aacd20" (UID: "5659da87-ec7e-4434-a21c-c33ac6aacd20"). InnerVolumeSpecName "kube-api-access-9tnxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:25.334746 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.334607 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-envoy" pod "5659da87-ec7e-4434-a21c-c33ac6aacd20" (UID: "5659da87-ec7e-4434-a21c-c33ac6aacd20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:25.334952 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.334930 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-token" pod "5659da87-ec7e-4434-a21c-c33ac6aacd20" (UID: "5659da87-ec7e-4434-a21c-c33ac6aacd20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:25.335039 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.335008 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-podinfo" pod "5659da87-ec7e-4434-a21c-c33ac6aacd20" (UID: "5659da87-ec7e-4434-a21c-c33ac6aacd20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:25.338546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.338530 4298 scope.go:122] "RemoveContainer" containerID="217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840" Nov 03 22:52:25.338940 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:25.338915 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840\": container with ID starting with 217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840 not found: ID does not exist" containerID="217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840" Nov 03 22:52:25.339017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.338947 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840"} err="failed to get container status \"217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840\": rpc error: code = NotFound desc = could not find container \"217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840\": container with ID starting with 217eb459facd0ec46571e026ab5a74ee1ef00a413047ffc8104a63f2a70f0840 not found: ID does not exist" Nov 03 22:52:25.339017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.338963 4298 scope.go:122] "RemoveContainer" containerID="cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55" Nov 03 22:52:25.339252 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:25.339233 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55\": container with ID starting with cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55 not found: ID does not exist" containerID="cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55" Nov 03 22:52:25.339320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.339260 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55"} err="failed to get container status \"cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55\": rpc error: code = NotFound desc = could not find container \"cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55\": container with ID starting with cebd917e16a2bec1f16717d3e0fbf1c982648f3bd596ded7b94c72777e967a55 not found: ID does not exist" Nov 03 22:52:25.434611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.434578 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:25.434772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.434616 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:25.434772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.434628 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tnxt\" (UniqueName: \"kubernetes.io/projected/5659da87-ec7e-4434-a21c-c33ac6aacd20-kube-api-access-9tnxt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:25.434772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.434653 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5659da87-ec7e-4434-a21c-c33ac6aacd20-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:25.490262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.490228 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv"] Nov 03 22:52:25.502482 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.502454 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt"] Nov 03 22:52:25.671252 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.671218 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:25.849806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.849768 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr"] Nov 03 22:52:25.853466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.853439 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b30a14-openshift-default-658cc46fc9-dvvmr"] Nov 03 22:52:25.870173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:25.870144 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" podUID="1022b4bf-b218-4df7-a0f4-951e5711896f" containerName="payload-pre-processing" containerID="cri-o://d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa" gracePeriod=25 Nov 03 22:52:26.135975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.135916 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:52:26.242354 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.242328 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1022b4bf-b218-4df7-a0f4-951e5711896f-kube-api-access-8z7r9\" (UniqueName: \"kubernetes.io/projected/1022b4bf-b218-4df7-a0f4-951e5711896f-kube-api-access-8z7r9\") pod \"1022b4bf-b218-4df7-a0f4-951e5711896f\" (UID: \"1022b4bf-b218-4df7-a0f4-951e5711896f\") " Nov 03 22:52:26.242490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.242421 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1022b4bf-b218-4df7-a0f4-951e5711896f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1022b4bf-b218-4df7-a0f4-951e5711896f-plugins-config-volume\") pod \"1022b4bf-b218-4df7-a0f4-951e5711896f\" (UID: \"1022b4bf-b218-4df7-a0f4-951e5711896f\") " Nov 03 22:52:26.242806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.242781 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1022b4bf-b218-4df7-a0f4-951e5711896f-plugins-config-volume" pod "1022b4bf-b218-4df7-a0f4-951e5711896f" (UID: "1022b4bf-b218-4df7-a0f4-951e5711896f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:26.244360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.244336 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1022b4bf-b218-4df7-a0f4-951e5711896f-kube-api-access-8z7r9" pod "1022b4bf-b218-4df7-a0f4-951e5711896f" (UID: "1022b4bf-b218-4df7-a0f4-951e5711896f"). InnerVolumeSpecName "kube-api-access-8z7r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:26.327632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.327597 4298 generic.go:356] "Generic (PLEG): container finished" podID="1aedc98d-0cc8-492e-ad70-6b7b5a661b1f" containerID="375dd0b04c1997bce72541d9498111c3996816b8280f6cb8d10cd680513add7c" exitCode=0 Nov 03 22:52:26.327632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.327629 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" event={"ID":"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f","Type":"ContainerDied","Data":"375dd0b04c1997bce72541d9498111c3996816b8280f6cb8d10cd680513add7c"} Nov 03 22:52:26.330066 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.330041 4298 generic.go:356] "Generic (PLEG): container finished" podID="1022b4bf-b218-4df7-a0f4-951e5711896f" containerID="d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa" exitCode=0 Nov 03 22:52:26.330184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.330083 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" Nov 03 22:52:26.330184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.330126 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" event={"ID":"1022b4bf-b218-4df7-a0f4-951e5711896f","Type":"ContainerDied","Data":"d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa"} Nov 03 22:52:26.330272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.330179 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d" event={"ID":"1022b4bf-b218-4df7-a0f4-951e5711896f","Type":"ContainerDied","Data":"ccf6d1acb2f846475eb74057ad59f05e0bc4ec33ec98994e7496bcbfc69b02f5"} Nov 03 22:52:26.330272 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.330207 4298 scope.go:122] "RemoveContainer" containerID="d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa" Nov 03 22:52:26.340786 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.340765 4298 scope.go:122] "RemoveContainer" containerID="d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa" Nov 03 22:52:26.341101 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:26.341064 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa\": container with ID starting with d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa not found: ID does not exist" containerID="d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa" Nov 03 22:52:26.341101 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.341089 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa"} err="failed to get container status \"d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa\": rpc error: code = NotFound desc = could not find container \"d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa\": container with ID starting with d7b7a8eb443476d17971781bf405d993ac65b39302f5cddcddfd72331c5666aa not found: ID does not exist" Nov 03 22:52:26.343723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.343697 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z7r9\" (UniqueName: \"kubernetes.io/projected/1022b4bf-b218-4df7-a0f4-951e5711896f-kube-api-access-8z7r9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:26.343836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.343727 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1022b4bf-b218-4df7-a0f4-951e5711896f-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:26.411517 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.411495 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:52:26.499663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.499617 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d"] Nov 03 22:52:26.503106 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.503086 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f7e6db-6dd7c88775-mfv8d"] Nov 03 22:52:26.546055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.546029 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-plugins-config-volume\") pod \"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f\" (UID: \"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f\") " Nov 03 22:52:26.546235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.546138 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-kube-api-access-rk2sm\" (UniqueName: \"kubernetes.io/projected/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-kube-api-access-rk2sm\") pod \"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f\" (UID: \"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f\") " Nov 03 22:52:26.546466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.546440 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-plugins-config-volume" pod "1aedc98d-0cc8-492e-ad70-6b7b5a661b1f" (UID: "1aedc98d-0cc8-492e-ad70-6b7b5a661b1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:26.548055 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.548024 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-kube-api-access-rk2sm" pod "1aedc98d-0cc8-492e-ad70-6b7b5a661b1f" (UID: "1aedc98d-0cc8-492e-ad70-6b7b5a661b1f"). InnerVolumeSpecName "kube-api-access-rk2sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:26.647930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.647902 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk2sm\" (UniqueName: \"kubernetes.io/projected/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-kube-api-access-rk2sm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:26.647930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:26.647928 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:27.134685 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:27.134630 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1022b4bf-b218-4df7-a0f4-951e5711896f" path="/var/lib/kubelet/pods/1022b4bf-b218-4df7-a0f4-951e5711896f/volumes" Nov 03 22:52:27.135153 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:27.135138 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" path="/var/lib/kubelet/pods/5659da87-ec7e-4434-a21c-c33ac6aacd20/volumes" Nov 03 22:52:27.226372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:27.226344 4298 scope.go:122] "RemoveContainer" containerID="d1de19adf5744252c62b757b7c32acd493536cf9e76da0bd7d519bf176b7dd41" Nov 03 22:52:27.235530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:27.235510 4298 scope.go:122] "RemoveContainer" containerID="375dd0b04c1997bce72541d9498111c3996816b8280f6cb8d10cd680513add7c" Nov 03 22:52:27.244621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:27.244599 4298 scope.go:122] "RemoveContainer" containerID="eb9941dba1e634189325fbb69e95dd7e11af82ae2fc23d9a3c0106cd8c151cbb" Nov 03 22:52:27.253363 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:27.253344 4298 scope.go:122] "RemoveContainer" containerID="f41c80503316333fa3fe09b4ca70f0837d5a57ee0c2e23a0089bdd2f34149adc" Nov 03 22:52:27.261495 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:27.261478 4298 scope.go:122] "RemoveContainer" containerID="0862162122b750a78d55383c2e4a4b261e68dd026be50ee7d68814597bdc064c" Nov 03 22:52:27.337054 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:27.337030 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" event={"ID":"1aedc98d-0cc8-492e-ad70-6b7b5a661b1f","Type":"ContainerDied","Data":"aa6df2b5ae630da80b7f12afb53f988e19749af9a1f799f7c2ab6be0048b4324"} Nov 03 22:52:27.337142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:27.337087 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8" Nov 03 22:52:27.562358 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:27.562328 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8"] Nov 03 22:52:27.565849 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:27.565823 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41bb2b-8db58b8c5-w6tx8"] Nov 03 22:52:28.725599 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:28.725554 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:29.146064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:29.145984 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aedc98d-0cc8-492e-ad70-6b7b5a661b1f" path="/var/lib/kubelet/pods/1aedc98d-0cc8-492e-ad70-6b7b5a661b1f/volumes" Nov 03 22:52:29.238385 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:29.238348 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:30.489218 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:30.489189 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:30.690163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:30.690126 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:32.101853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:32.101812 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:32.540578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:32.540550 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:32.682747 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:32.682711 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9"] Nov 03 22:52:32.835094 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:32.834989 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:33.019732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:33.019701 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:33.214527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:33.214497 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:33.534930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:33.534843 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:34.687277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:34.687238 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:35.083488 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:35.083449 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:35.325481 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:35.325445 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:35.505783 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:35.505750 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:35.614596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:35.614556 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k"] Nov 03 22:52:35.665103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:35.665066 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-fac3b6-2js5k"] Nov 03 22:52:36.038474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:36.038431 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:37.134328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:37.134294 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbd19d3-a6c5-4036-8af3-c3cc75926b3b" path="/var/lib/kubelet/pods/bbbd19d3-a6c5-4036-8af3-c3cc75926b3b/volumes" Nov 03 22:52:37.683326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:37.683278 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" podUID="ecea227c-1b19-4a6c-b115-77d3987557f8" containerName="payload-pre-processing" containerID="cri-o://0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0" gracePeriod=25 Nov 03 22:52:37.742626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:37.742596 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:37.928604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:37.928577 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:37.947049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:37.947023 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:52:37.974936 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:37.974907 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecea227c-1b19-4a6c-b115-77d3987557f8-kube-api-access-tf7n6\" (UniqueName: \"kubernetes.io/projected/ecea227c-1b19-4a6c-b115-77d3987557f8-kube-api-access-tf7n6\") pod \"ecea227c-1b19-4a6c-b115-77d3987557f8\" (UID: \"ecea227c-1b19-4a6c-b115-77d3987557f8\") " Nov 03 22:52:37.975077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:37.974949 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecea227c-1b19-4a6c-b115-77d3987557f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecea227c-1b19-4a6c-b115-77d3987557f8-plugins-config-volume\") pod \"ecea227c-1b19-4a6c-b115-77d3987557f8\" (UID: \"ecea227c-1b19-4a6c-b115-77d3987557f8\") " Nov 03 22:52:37.975322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:37.975302 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecea227c-1b19-4a6c-b115-77d3987557f8-plugins-config-volume" pod "ecea227c-1b19-4a6c-b115-77d3987557f8" (UID: "ecea227c-1b19-4a6c-b115-77d3987557f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:37.977026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:37.977006 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecea227c-1b19-4a6c-b115-77d3987557f8-kube-api-access-tf7n6" pod "ecea227c-1b19-4a6c-b115-77d3987557f8" (UID: "ecea227c-1b19-4a6c-b115-77d3987557f8"). InnerVolumeSpecName "kube-api-access-tf7n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:38.076779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:38.076746 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf7n6\" (UniqueName: \"kubernetes.io/projected/ecea227c-1b19-4a6c-b115-77d3987557f8-kube-api-access-tf7n6\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:38.076779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:38.076774 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecea227c-1b19-4a6c-b115-77d3987557f8-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:38.403451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:38.403364 4298 generic.go:356] "Generic (PLEG): container finished" podID="ecea227c-1b19-4a6c-b115-77d3987557f8" containerID="0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0" exitCode=0 Nov 03 22:52:38.403451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:38.403412 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" Nov 03 22:52:38.403984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:38.403452 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" event={"ID":"ecea227c-1b19-4a6c-b115-77d3987557f8","Type":"ContainerDied","Data":"0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0"} Nov 03 22:52:38.403984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:38.403505 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9" event={"ID":"ecea227c-1b19-4a6c-b115-77d3987557f8","Type":"ContainerDied","Data":"a634c7408fda99f11b3c3c4f79b9a596c2119c2167051aeb24e4b9b5c8eefb44"} Nov 03 22:52:38.403984 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:38.403531 4298 scope.go:122] "RemoveContainer" containerID="0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0" Nov 03 22:52:38.413472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:38.413456 4298 scope.go:122] "RemoveContainer" containerID="0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0" Nov 03 22:52:38.413786 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:38.413753 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0\": container with ID starting with 0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0 not found: ID does not exist" containerID="0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0" Nov 03 22:52:38.413866 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:38.413798 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0"} err="failed to get container status \"0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0\": rpc error: code = NotFound desc = could not find container \"0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0\": container with ID starting with 0b9af6eab92086d8822fc296b9a907f2a450d2889aaa0dd78425fb9a7eda2ee0 not found: ID does not exist" Nov 03 22:52:38.568502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:38.568462 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9"] Nov 03 22:52:38.618620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:38.618583 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-906673-7556857999-qqfc9"] Nov 03 22:52:39.133414 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:39.133380 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecea227c-1b19-4a6c-b115-77d3987557f8" path="/var/lib/kubelet/pods/ecea227c-1b19-4a6c-b115-77d3987557f8/volumes" Nov 03 22:52:39.864301 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:39.864266 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:40.513564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.513532 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" containerID="cri-o://7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8" gracePeriod=45 Nov 03 22:52:40.523838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.523812 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:40.526028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.526004 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" containerID="cri-o://9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c" gracePeriod=45 Nov 03 22:52:40.829877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.829847 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:52:40.845097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.845078 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:52:40.910242 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910209 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-home\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-home\") pod \"d48a9429-f937-41c1-b434-d7fbef05ae0c\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910269 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-model-cache\") pod \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910298 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-tmp-dir\") pod \"d48a9429-f937-41c1-b434-d7fbef05ae0c\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910327 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48a9429-f937-41c1-b434-d7fbef05ae0c-kube-api-access-s48hv\" (UniqueName: \"kubernetes.io/projected/d48a9429-f937-41c1-b434-d7fbef05ae0c-kube-api-access-s48hv\") pod \"d48a9429-f937-41c1-b434-d7fbef05ae0c\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910369 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-dshm\") pod \"d48a9429-f937-41c1-b434-d7fbef05ae0c\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910484 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tls-certs\") pod \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910516 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48a9429-f937-41c1-b434-d7fbef05ae0c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d48a9429-f937-41c1-b434-d7fbef05ae0c-tls-certs\") pod \"d48a9429-f937-41c1-b434-d7fbef05ae0c\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910531 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-home" pod "d48a9429-f937-41c1-b434-d7fbef05ae0c" (UID: "d48a9429-f937-41c1-b434-d7fbef05ae0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910551 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-home\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-home\") pod \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910579 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tmp-dir\") pod \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910580 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-model-cache" pod "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" (UID: "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910602 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-tmp-dir" pod "d48a9429-f937-41c1-b434-d7fbef05ae0c" (UID: "d48a9429-f937-41c1-b434-d7fbef05ae0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.910699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910635 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-dshm\") pod \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " Nov 03 22:52:40.911395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910716 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-model-cache\") pod \"d48a9429-f937-41c1-b434-d7fbef05ae0c\" (UID: \"d48a9429-f937-41c1-b434-d7fbef05ae0c\") " Nov 03 22:52:40.911395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910764 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-kube-api-access-6rktz\" (UniqueName: \"kubernetes.io/projected/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-kube-api-access-6rktz\") pod \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\" (UID: \"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2\") " Nov 03 22:52:40.911395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910861 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-home" pod "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" (UID: "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.911395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.910913 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tmp-dir" pod "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" (UID: "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.911395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.911055 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-model-cache" pod "d48a9429-f937-41c1-b434-d7fbef05ae0c" (UID: "d48a9429-f937-41c1-b434-d7fbef05ae0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.911395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.911347 4298 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-home\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.911395 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.911396 4298 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-model-cache\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.911998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.911424 4298 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-tmp-dir\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.911998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.911440 4298 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-home\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.911998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.911494 4298 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tmp-dir\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.911998 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.911508 4298 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-model-cache\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.913185 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.913157 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-dshm" pod "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" (UID: "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.913393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.913367 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tls-certs" pod "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" (UID: "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:40.913393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.913378 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-kube-api-access-6rktz" pod "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" (UID: "1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2"). InnerVolumeSpecName "kube-api-access-6rktz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:40.913532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.913404 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48a9429-f937-41c1-b434-d7fbef05ae0c-kube-api-access-s48hv" pod "d48a9429-f937-41c1-b434-d7fbef05ae0c" (UID: "d48a9429-f937-41c1-b434-d7fbef05ae0c"). InnerVolumeSpecName "kube-api-access-s48hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:40.913532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.913404 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-dshm" pod "d48a9429-f937-41c1-b434-d7fbef05ae0c" (UID: "d48a9429-f937-41c1-b434-d7fbef05ae0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.913532 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:40.913497 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48a9429-f937-41c1-b434-d7fbef05ae0c-tls-certs" pod "d48a9429-f937-41c1-b434-d7fbef05ae0c" (UID: "d48a9429-f937-41c1-b434-d7fbef05ae0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:41.012812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.012769 4298 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-tls-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.012812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.012798 4298 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48a9429-f937-41c1-b434-d7fbef05ae0c-tls-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.012812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.012808 4298 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-dshm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.012812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.012818 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rktz\" (UniqueName: \"kubernetes.io/projected/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2-kube-api-access-6rktz\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.013178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.012829 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s48hv\" (UniqueName: \"kubernetes.io/projected/d48a9429-f937-41c1-b434-d7fbef05ae0c-kube-api-access-s48hv\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.013178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.012837 4298 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48a9429-f937-41c1-b434-d7fbef05ae0c-dshm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.407326 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.407232 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:41.423163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.423132 4298 generic.go:356] "Generic (PLEG): container finished" podID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerID="7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8" exitCode=0 Nov 03 22:52:41.423310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.423175 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" Nov 03 22:52:41.423310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.423218 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" event={"ID":"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2","Type":"ContainerDied","Data":"7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8"} Nov 03 22:52:41.423310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.423265 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv" event={"ID":"1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2","Type":"ContainerDied","Data":"042d9e550b8b72e1ac20b7bc1b038c6441e5c0cf13657ac84a06194000330db0"} Nov 03 22:52:41.423310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.423288 4298 scope.go:122] "RemoveContainer" containerID="7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8" Nov 03 22:52:41.424925 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.424901 4298 generic.go:356] "Generic (PLEG): container finished" podID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerID="9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c" exitCode=0 Nov 03 22:52:41.425044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.424932 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" event={"ID":"d48a9429-f937-41c1-b434-d7fbef05ae0c","Type":"ContainerDied","Data":"9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c"} Nov 03 22:52:41.425044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.424985 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" event={"ID":"d48a9429-f937-41c1-b434-d7fbef05ae0c","Type":"ContainerDied","Data":"c1e965d5af967dd3b6de24399963677f2a02bf885475c2f5538d288878cfce80"} Nov 03 22:52:41.425044 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.424954 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt" Nov 03 22:52:41.433017 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.432995 4298 scope.go:122] "RemoveContainer" containerID="7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8" Nov 03 22:52:41.433269 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:41.433251 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8\": container with ID starting with 7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8 not found: ID does not exist" containerID="7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8" Nov 03 22:52:41.433347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.433280 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8"} err="failed to get container status \"7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8\": rpc error: code = NotFound desc = could not find container \"7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8\": container with ID starting with 7127772c52fe30827534130c4565084739c1c2471306a784866e72ca31eb56c8 not found: ID does not exist" Nov 03 22:52:41.433347 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.433303 4298 scope.go:122] "RemoveContainer" containerID="9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c" Nov 03 22:52:41.441169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.441152 4298 scope.go:122] "RemoveContainer" containerID="9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c" Nov 03 22:52:41.441434 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:41.441414 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c\": container with ID starting with 9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c not found: ID does not exist" containerID="9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c" Nov 03 22:52:41.441492 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.441440 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c"} err="failed to get container status \"9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c\": rpc error: code = NotFound desc = could not find container \"9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c\": container with ID starting with 9517fa9998484b0e3c6601a2d021c5d7aa1e38a817efb48fbe868dc10753155c not found: ID does not exist" Nov 03 22:52:41.592609 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.592580 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt"] Nov 03 22:52:41.596299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.596274 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-906673/simulator-w2-906673-kserve-675c5c9bfb-hlwdt"] Nov 03 22:52:41.650375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.650347 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:41.705341 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.705312 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv"] Nov 03 22:52:41.709146 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.709115 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-906673/premium-w2-906673-kserve-f4d448dbf-9xnfv"] Nov 03 22:52:41.832500 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.832429 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:41.884997 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.884954 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7"] Nov 03 22:52:41.888097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:41.888078 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-41bb2b-nhhd7"] Nov 03 22:52:42.029187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:42.029101 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:42.234875 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:42.234841 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh"] Nov 03 22:52:42.235120 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:42.235093 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" containerID="cri-o://5c791e8d595cbe2021c6e1941f98f840af81fa9dbe30c0375b521fb3b2521dfc" gracePeriod=30 Nov 03 22:52:42.313010 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:42.312907 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:42.494816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:42.494779 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:43.077904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.077868 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7"] Nov 03 22:52:43.078280 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078264 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" Nov 03 22:52:43.078280 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078274 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" Nov 03 22:52:43.078361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078285 4298 cpu_manager.go:414] "containerMap: removing container" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" Nov 03 22:52:43.078361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078291 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" Nov 03 22:52:43.078361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078303 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1aedc98d-0cc8-492e-ad70-6b7b5a661b1f" containerName="payload-pre-processing" Nov 03 22:52:43.078361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078311 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aedc98d-0cc8-492e-ad70-6b7b5a661b1f" containerName="payload-pre-processing" Nov 03 22:52:43.078361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078326 4298 cpu_manager.go:414] "containerMap: removing container" podUID="af323b31-3c27-4a31-87e8-94a5ae96aafc" containerName="revoke-keys" Nov 03 22:52:43.078361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078332 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af323b31-3c27-4a31-87e8-94a5ae96aafc" containerName="revoke-keys" Nov 03 22:52:43.078361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078348 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1022b4bf-b218-4df7-a0f4-951e5711896f" containerName="payload-pre-processing" Nov 03 22:52:43.078361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078352 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1022b4bf-b218-4df7-a0f4-951e5711896f" containerName="payload-pre-processing" Nov 03 22:52:43.078361 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078364 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ecea227c-1b19-4a6c-b115-77d3987557f8" containerName="payload-pre-processing" Nov 03 22:52:43.078657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078368 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecea227c-1b19-4a6c-b115-77d3987557f8" containerName="payload-pre-processing" Nov 03 22:52:43.078657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078380 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" Nov 03 22:52:43.078657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078384 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" Nov 03 22:52:43.078657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078442 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" containerName="main" Nov 03 22:52:43.078657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078447 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" Nov 03 22:52:43.078657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078453 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aedc98d-0cc8-492e-ad70-6b7b5a661b1f" containerName="payload-pre-processing" Nov 03 22:52:43.078657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078462 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="af323b31-3c27-4a31-87e8-94a5ae96aafc" containerName="revoke-keys" Nov 03 22:52:43.078657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078469 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1022b4bf-b218-4df7-a0f4-951e5711896f" containerName="payload-pre-processing" Nov 03 22:52:43.078657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078475 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecea227c-1b19-4a6c-b115-77d3987557f8" containerName="payload-pre-processing" Nov 03 22:52:43.078657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078480 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" Nov 03 22:52:43.078657 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.078486 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" containerName="main" Nov 03 22:52:43.082904 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.082887 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:43.134758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.134728 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2" path="/var/lib/kubelet/pods/1cbb21e0-adc8-4a3f-ae50-ebd48828d4d2/volumes" Nov 03 22:52:43.135170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.135156 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48a9429-f937-41c1-b434-d7fbef05ae0c" path="/var/lib/kubelet/pods/d48a9429-f937-41c1-b434-d7fbef05ae0c/volumes" Nov 03 22:52:43.135723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.135705 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1ea316-d7d5-467f-b99c-8b3d8f15bcae" path="/var/lib/kubelet/pods/ff1ea316-d7d5-467f-b99c-8b3d8f15bcae/volumes" Nov 03 22:52:43.138163 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.138133 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqz5v\" (UniqueName: \"kubernetes.io/projected/5210e1e3-1c7d-490d-be3f-541812789f02-kube-api-access-lqz5v\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:43.138302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.138258 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:43.138369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.138350 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:43.140435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.140413 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7"] Nov 03 22:52:43.185856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.185826 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:52:43.186043 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.185974 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:52:43.239554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.239515 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqz5v\" (UniqueName: \"kubernetes.io/projected/5210e1e3-1c7d-490d-be3f-541812789f02-kube-api-access-lqz5v\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:43.239797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.239580 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:43.239797 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.239756 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:43.240398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.240378 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:43.241855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.241837 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:43.297065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.297033 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqz5v\" (UniqueName: \"kubernetes.io/projected/5210e1e3-1c7d-490d-be3f-541812789f02-kube-api-access-lqz5v\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:43.396049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.395970 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:43.764179 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:52:43.764148 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5210e1e3_1c7d_490d_be3f_541812789f02.slice/crio-666222cf91aba4304b3ea3be29b74ca8c60b9a4ccfdd2169e3863f8dfa771bc5 WatchSource:0}: Error finding container 666222cf91aba4304b3ea3be29b74ca8c60b9a4ccfdd2169e3863f8dfa771bc5: Status 404 returned error can't find the container with id 666222cf91aba4304b3ea3be29b74ca8c60b9a4ccfdd2169e3863f8dfa771bc5 Nov 03 22:52:43.767400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:43.767374 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7"] Nov 03 22:52:44.446271 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:44.446227 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" event={"ID":"5210e1e3-1c7d-490d-be3f-541812789f02","Type":"ContainerStarted","Data":"ac949e45c8f276449be501c59346ef260a30dd1ebdd01a1b372d5f89ccdc40d9"} Nov 03 22:52:44.446698 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:44.446296 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" event={"ID":"5210e1e3-1c7d-490d-be3f-541812789f02","Type":"ContainerStarted","Data":"666222cf91aba4304b3ea3be29b74ca8c60b9a4ccfdd2169e3863f8dfa771bc5"} Nov 03 22:52:44.562975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:44.562921 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" podStartSLOduration=1.56290697 podStartE2EDuration="1.56290697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:44.559599835 +0000 UTC m=+2721.540333489" watchObservedRunningTime="2026-11-03 22:52:44.56290697 +0000 UTC m=+2721.543640623" Nov 03 22:52:45.454560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:45.454523 4298 generic.go:356] "Generic (PLEG): container finished" podID="5210e1e3-1c7d-490d-be3f-541812789f02" containerID="ac949e45c8f276449be501c59346ef260a30dd1ebdd01a1b372d5f89ccdc40d9" exitCode=0 Nov 03 22:52:45.455019 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:45.454618 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" event={"ID":"5210e1e3-1c7d-490d-be3f-541812789f02","Type":"ContainerDied","Data":"ac949e45c8f276449be501c59346ef260a30dd1ebdd01a1b372d5f89ccdc40d9"} Nov 03 22:52:46.226183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.226152 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5"] Nov 03 22:52:46.226603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.226590 4298 cpu_manager.go:414] "containerMap: removing container" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" Nov 03 22:52:46.226687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.226605 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5659da87-ec7e-4434-a21c-c33ac6aacd20" containerName="istio-proxy" Nov 03 22:52:46.229896 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.229881 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:46.286611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.286575 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5"] Nov 03 22:52:46.374750 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.374710 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de649233-8jzt5\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:46.374944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.374758 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5sr\" (UniqueName: \"kubernetes.io/projected/1accc7f5-0255-43de-8ce1-5f892f8ff521-kube-api-access-bp5sr\") pod \"maas-api-revoke-sub-de649233-8jzt5\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:46.374944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.374861 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de649233-8jzt5\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:46.476158 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.476114 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de649233-8jzt5\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:46.476699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.476177 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5sr\" (UniqueName: \"kubernetes.io/projected/1accc7f5-0255-43de-8ce1-5f892f8ff521-kube-api-access-bp5sr\") pod \"maas-api-revoke-sub-de649233-8jzt5\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:46.476699 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.476237 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de649233-8jzt5\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:46.477098 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.477073 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de649233-8jzt5\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:46.478826 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.478802 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de649233-8jzt5\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:46.499040 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.499009 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p"] Nov 03 22:52:46.502853 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.502835 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:46.547493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.547459 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5sr\" (UniqueName: \"kubernetes.io/projected/1accc7f5-0255-43de-8ce1-5f892f8ff521-kube-api-access-bp5sr\") pod \"maas-api-revoke-sub-de649233-8jzt5\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:46.559973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.559684 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p"] Nov 03 22:52:46.577842 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.577817 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hszz\" (UniqueName: \"kubernetes.io/projected/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-kube-api-access-6hszz\") pod \"maas-api-revoke-sub-145293ea-wcp5p\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:46.577966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.577925 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-145293ea-wcp5p\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:46.578013 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.577985 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-145293ea-wcp5p\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:46.598807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.598781 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:46.678726 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.678688 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-openshift-ca\") pod \"5210e1e3-1c7d-490d-be3f-541812789f02\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " Nov 03 22:52:46.678944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.678761 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-xks-ca\") pod \"5210e1e3-1c7d-490d-be3f-541812789f02\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " Nov 03 22:52:46.678944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.678795 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5210e1e3-1c7d-490d-be3f-541812789f02-kube-api-access-lqz5v\" (UniqueName: \"kubernetes.io/projected/5210e1e3-1c7d-490d-be3f-541812789f02-kube-api-access-lqz5v\") pod \"5210e1e3-1c7d-490d-be3f-541812789f02\" (UID: \"5210e1e3-1c7d-490d-be3f-541812789f02\") " Nov 03 22:52:46.678944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.678936 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-145293ea-wcp5p\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:46.679190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.679059 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hszz\" (UniqueName: \"kubernetes.io/projected/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-kube-api-access-6hszz\") pod \"maas-api-revoke-sub-145293ea-wcp5p\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:46.679190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.679066 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-openshift-ca" pod "5210e1e3-1c7d-490d-be3f-541812789f02" (UID: "5210e1e3-1c7d-490d-be3f-541812789f02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:46.679190 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.679167 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-145293ea-wcp5p\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:46.679362 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.679296 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:46.679615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.679592 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-145293ea-wcp5p\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:46.681200 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.681171 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-xks-ca" pod "5210e1e3-1c7d-490d-be3f-541812789f02" (UID: "5210e1e3-1c7d-490d-be3f-541812789f02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:46.681428 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.681403 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5210e1e3-1c7d-490d-be3f-541812789f02-kube-api-access-lqz5v" pod "5210e1e3-1c7d-490d-be3f-541812789f02" (UID: "5210e1e3-1c7d-490d-be3f-541812789f02"). InnerVolumeSpecName "kube-api-access-lqz5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:46.681568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.681548 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-145293ea-wcp5p\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:46.736574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.736501 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hszz\" (UniqueName: \"kubernetes.io/projected/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-kube-api-access-6hszz\") pod \"maas-api-revoke-sub-145293ea-wcp5p\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:46.780079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.780045 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqz5v\" (UniqueName: \"kubernetes.io/projected/5210e1e3-1c7d-490d-be3f-541812789f02-kube-api-access-lqz5v\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:46.780079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.780076 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5210e1e3-1c7d-490d-be3f-541812789f02-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:46.816855 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.816815 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:46.842393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:46.842364 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:47.196299 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.196265 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p"] Nov 03 22:52:47.216268 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:52:47.216237 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cc5baa_8a68_442b_9ceb_1ce6016e2ca6.slice/crio-ed59b6b8cae992d0113ba33f75a4f146279a561a07c1c919ee842adc28baef16 WatchSource:0}: Error finding container ed59b6b8cae992d0113ba33f75a4f146279a561a07c1c919ee842adc28baef16: Status 404 returned error can't find the container with id ed59b6b8cae992d0113ba33f75a4f146279a561a07c1c919ee842adc28baef16 Nov 03 22:52:47.244753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.244725 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5"] Nov 03 22:52:47.246320 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:52:47.246297 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1accc7f5_0255_43de_8ce1_5f892f8ff521.slice/crio-d445dbb0f360c9df0c51a1ce792761744bd830b37786e1ca6c4de38d72434b5c WatchSource:0}: Error finding container d445dbb0f360c9df0c51a1ce792761744bd830b37786e1ca6c4de38d72434b5c: Status 404 returned error can't find the container with id d445dbb0f360c9df0c51a1ce792761744bd830b37786e1ca6c4de38d72434b5c Nov 03 22:52:47.520208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.520171 4298 generic.go:356] "Generic (PLEG): container finished" podID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerID="5c791e8d595cbe2021c6e1941f98f840af81fa9dbe30c0375b521fb3b2521dfc" exitCode=0 Nov 03 22:52:47.521228 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.521194 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" event={"ID":"1ef2fc6c-a501-4a62-adcf-879a017e7bd8","Type":"ContainerDied","Data":"5c791e8d595cbe2021c6e1941f98f840af81fa9dbe30c0375b521fb3b2521dfc"} Nov 03 22:52:47.521372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.521274 4298 scope.go:122] "RemoveContainer" containerID="abe257aca0704ed70808099dd8dedbeb68c9e10d08f0f74fc55f4d744f03a57c" Nov 03 22:52:47.523401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.523108 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" event={"ID":"5210e1e3-1c7d-490d-be3f-541812789f02","Type":"ContainerDied","Data":"666222cf91aba4304b3ea3be29b74ca8c60b9a4ccfdd2169e3863f8dfa771bc5"} Nov 03 22:52:47.523401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.523130 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7" Nov 03 22:52:47.523401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.523157 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="666222cf91aba4304b3ea3be29b74ca8c60b9a4ccfdd2169e3863f8dfa771bc5" Nov 03 22:52:47.525523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.525489 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" event={"ID":"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6","Type":"ContainerStarted","Data":"ed59b6b8cae992d0113ba33f75a4f146279a561a07c1c919ee842adc28baef16"} Nov 03 22:52:47.527503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.527450 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:47.528308 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.528249 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" event={"ID":"1accc7f5-0255-43de-8ce1-5f892f8ff521","Type":"ContainerStarted","Data":"d445dbb0f360c9df0c51a1ce792761744bd830b37786e1ca6c4de38d72434b5c"} Nov 03 22:52:47.552473 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:47.552445 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:52:47.563965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.563940 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:52:47.709810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.709777 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-kube-api-access-cqxwd\" (UniqueName: \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-kube-api-access-cqxwd\") pod \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " Nov 03 22:52:47.710030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.709867 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-envoy\") pod \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " Nov 03 22:52:47.710030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.709913 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-credential-socket\") pod \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " Nov 03 22:52:47.710030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.709965 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-podinfo\") pod \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " Nov 03 22:52:47.710030 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.709989 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istiod-ca-cert\") pod \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " Nov 03 22:52:47.710691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710038 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-data\") pod \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " Nov 03 22:52:47.710691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710152 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-certs\") pod \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " Nov 03 22:52:47.710691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710203 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-socket\") pod \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " Nov 03 22:52:47.710691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710232 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-token\" (UniqueName: \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-token\") pod \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\" (UID: \"1ef2fc6c-a501-4a62-adcf-879a017e7bd8\") " Nov 03 22:52:47.710691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710423 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istiod-ca-cert" pod "1ef2fc6c-a501-4a62-adcf-879a017e7bd8" (UID: "1ef2fc6c-a501-4a62-adcf-879a017e7bd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:47.710691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710443 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-certs" pod "1ef2fc6c-a501-4a62-adcf-879a017e7bd8" (UID: "1ef2fc6c-a501-4a62-adcf-879a017e7bd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:47.710691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710403 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-data" pod "1ef2fc6c-a501-4a62-adcf-879a017e7bd8" (UID: "1ef2fc6c-a501-4a62-adcf-879a017e7bd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:47.710691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710662 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:47.710691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710670 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-socket" pod "1ef2fc6c-a501-4a62-adcf-879a017e7bd8" (UID: "1ef2fc6c-a501-4a62-adcf-879a017e7bd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:47.710691 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710683 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:47.711149 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710718 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:47.711149 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.710878 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-credential-socket" pod "1ef2fc6c-a501-4a62-adcf-879a017e7bd8" (UID: "1ef2fc6c-a501-4a62-adcf-879a017e7bd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:47.712445 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.712413 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-kube-api-access-cqxwd" pod "1ef2fc6c-a501-4a62-adcf-879a017e7bd8" (UID: "1ef2fc6c-a501-4a62-adcf-879a017e7bd8"). InnerVolumeSpecName "kube-api-access-cqxwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:47.712561 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.712476 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-token" pod "1ef2fc6c-a501-4a62-adcf-879a017e7bd8" (UID: "1ef2fc6c-a501-4a62-adcf-879a017e7bd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:47.712614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.712573 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-podinfo" pod "1ef2fc6c-a501-4a62-adcf-879a017e7bd8" (UID: "1ef2fc6c-a501-4a62-adcf-879a017e7bd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:47.712614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.712584 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-envoy" pod "1ef2fc6c-a501-4a62-adcf-879a017e7bd8" (UID: "1ef2fc6c-a501-4a62-adcf-879a017e7bd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:47.812213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.812179 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:47.812213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.812209 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:47.812213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.812221 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqxwd\" (UniqueName: \"kubernetes.io/projected/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-kube-api-access-cqxwd\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:47.812470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.812231 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:47.812470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.812240 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:47.812470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:47.812248 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef2fc6c-a501-4a62-adcf-879a017e7bd8-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:48.535776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.535736 4298 generic.go:356] "Generic (PLEG): container finished" podID="b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6" containerID="97c27e090c5212af3f4c343227d529565fde62e85e532dcbc403d3625ee5e469" exitCode=0 Nov 03 22:52:48.536230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.535826 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" event={"ID":"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6","Type":"ContainerDied","Data":"97c27e090c5212af3f4c343227d529565fde62e85e532dcbc403d3625ee5e469"} Nov 03 22:52:48.537420 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.537390 4298 generic.go:356] "Generic (PLEG): container finished" podID="1accc7f5-0255-43de-8ce1-5f892f8ff521" containerID="4d0f39741c27f2b0997343d2dc9f2579d60fb1e633113d0b33b44cbf1727c074" exitCode=0 Nov 03 22:52:48.537541 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.537473 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" event={"ID":"1accc7f5-0255-43de-8ce1-5f892f8ff521","Type":"ContainerDied","Data":"4d0f39741c27f2b0997343d2dc9f2579d60fb1e633113d0b33b44cbf1727c074"} Nov 03 22:52:48.538979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.538957 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" event={"ID":"1ef2fc6c-a501-4a62-adcf-879a017e7bd8","Type":"ContainerDied","Data":"2955ec7f5cd3a6e062edf271a1ddc0dcd21053da60d425139071a4725c2ad4d5"} Nov 03 22:52:48.538979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.538969 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh" Nov 03 22:52:48.539121 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.539010 4298 scope.go:122] "RemoveContainer" containerID="5c791e8d595cbe2021c6e1941f98f840af81fa9dbe30c0375b521fb3b2521dfc" Nov 03 22:52:48.684805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.684767 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:48.764615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.764513 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh"] Nov 03 22:52:48.814846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.814813 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-fac3b6-openshift-default-58b8d94d67-cpldh"] Nov 03 22:52:48.875848 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.875811 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:48.938557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.938525 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4"] Nov 03 22:52:48.938861 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:48.938833 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerName="istio-proxy" containerID="cri-o://4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb" gracePeriod=30 Nov 03 22:52:49.130906 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.130817 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:49.133575 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.133550 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" path="/var/lib/kubelet/pods/1ef2fc6c-a501-4a62-adcf-879a017e7bd8/volumes" Nov 03 22:52:49.699736 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.699706 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:49.701703 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.701686 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:49.834985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.834898 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-openshift-ca\") pod \"1accc7f5-0255-43de-8ce1-5f892f8ff521\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " Nov 03 22:52:49.834985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.834956 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-xks-ca\") pod \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " Nov 03 22:52:49.835226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.835009 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1accc7f5-0255-43de-8ce1-5f892f8ff521-kube-api-access-bp5sr\" (UniqueName: \"kubernetes.io/projected/1accc7f5-0255-43de-8ce1-5f892f8ff521-kube-api-access-bp5sr\") pod \"1accc7f5-0255-43de-8ce1-5f892f8ff521\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " Nov 03 22:52:49.835226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.835091 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-xks-ca\") pod \"1accc7f5-0255-43de-8ce1-5f892f8ff521\" (UID: \"1accc7f5-0255-43de-8ce1-5f892f8ff521\") " Nov 03 22:52:49.835226 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.835164 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-openshift-ca\") pod \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " Nov 03 22:52:49.835389 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.835317 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-openshift-ca" pod "1accc7f5-0255-43de-8ce1-5f892f8ff521" (UID: "1accc7f5-0255-43de-8ce1-5f892f8ff521"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:49.835463 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.835439 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-kube-api-access-6hszz\" (UniqueName: \"kubernetes.io/projected/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-kube-api-access-6hszz\") pod \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\" (UID: \"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6\") " Nov 03 22:52:49.835578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.835556 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-openshift-ca" pod "b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6" (UID: "b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:49.835883 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.835863 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:49.835962 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.835890 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:49.837348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.837318 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-xks-ca" pod "b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6" (UID: "b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:49.837590 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.837566 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-kube-api-access-6hszz" pod "b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6" (UID: "b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6"). InnerVolumeSpecName "kube-api-access-6hszz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:49.837735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.837701 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1accc7f5-0255-43de-8ce1-5f892f8ff521-kube-api-access-bp5sr" pod "1accc7f5-0255-43de-8ce1-5f892f8ff521" (UID: "1accc7f5-0255-43de-8ce1-5f892f8ff521"). InnerVolumeSpecName "kube-api-access-bp5sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:49.838086 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.838063 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-xks-ca" pod "1accc7f5-0255-43de-8ce1-5f892f8ff521" (UID: "1accc7f5-0255-43de-8ce1-5f892f8ff521"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:49.936840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.936803 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1accc7f5-0255-43de-8ce1-5f892f8ff521-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:49.936840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.936842 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hszz\" (UniqueName: \"kubernetes.io/projected/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-kube-api-access-6hszz\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:49.937096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.936860 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:49.937096 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:49.936874 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp5sr\" (UniqueName: \"kubernetes.io/projected/1accc7f5-0255-43de-8ce1-5f892f8ff521-kube-api-access-bp5sr\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:50.553183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:50.553149 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" Nov 03 22:52:50.553374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:50.553154 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p" event={"ID":"b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6","Type":"ContainerDied","Data":"ed59b6b8cae992d0113ba33f75a4f146279a561a07c1c919ee842adc28baef16"} Nov 03 22:52:50.553374 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:50.553270 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed59b6b8cae992d0113ba33f75a4f146279a561a07c1c919ee842adc28baef16" Nov 03 22:52:50.554574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:50.554555 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" Nov 03 22:52:50.554709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:50.554592 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5" event={"ID":"1accc7f5-0255-43de-8ce1-5f892f8ff521","Type":"ContainerDied","Data":"d445dbb0f360c9df0c51a1ce792761744bd830b37786e1ca6c4de38d72434b5c"} Nov 03 22:52:50.554709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:50.554638 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d445dbb0f360c9df0c51a1ce792761744bd830b37786e1ca6c4de38d72434b5c" Nov 03 22:52:52.426923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:52.426884 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:54.223484 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.223458 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:52:54.286250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286213 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-podinfo\") pod \"afee2ffe-35e2-4903-849c-fb37e18b8b70\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " Nov 03 22:52:54.286456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286259 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-token\" (UniqueName: \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-token\") pod \"afee2ffe-35e2-4903-849c-fb37e18b8b70\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " Nov 03 22:52:54.286456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286333 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-credential-socket\") pod \"afee2ffe-35e2-4903-849c-fb37e18b8b70\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " Nov 03 22:52:54.286456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286363 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-socket\") pod \"afee2ffe-35e2-4903-849c-fb37e18b8b70\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " Nov 03 22:52:54.286456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286402 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-certs\") pod \"afee2ffe-35e2-4903-849c-fb37e18b8b70\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " Nov 03 22:52:54.286456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286443 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-data\") pod \"afee2ffe-35e2-4903-849c-fb37e18b8b70\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " Nov 03 22:52:54.286778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286490 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-kube-api-access-pmtfx\" (UniqueName: \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-kube-api-access-pmtfx\") pod \"afee2ffe-35e2-4903-849c-fb37e18b8b70\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " Nov 03 22:52:54.286778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286525 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afee2ffe-35e2-4903-849c-fb37e18b8b70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afee2ffe-35e2-4903-849c-fb37e18b8b70-istiod-ca-cert\") pod \"afee2ffe-35e2-4903-849c-fb37e18b8b70\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " Nov 03 22:52:54.286778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286552 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-envoy\") pod \"afee2ffe-35e2-4903-849c-fb37e18b8b70\" (UID: \"afee2ffe-35e2-4903-849c-fb37e18b8b70\") " Nov 03 22:52:54.286778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286684 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-credential-socket" pod "afee2ffe-35e2-4903-849c-fb37e18b8b70" (UID: "afee2ffe-35e2-4903-849c-fb37e18b8b70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:54.286778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286751 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-socket" pod "afee2ffe-35e2-4903-849c-fb37e18b8b70" (UID: "afee2ffe-35e2-4903-849c-fb37e18b8b70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:54.286778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286768 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-data" pod "afee2ffe-35e2-4903-849c-fb37e18b8b70" (UID: "afee2ffe-35e2-4903-849c-fb37e18b8b70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:54.287117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286861 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-certs" pod "afee2ffe-35e2-4903-849c-fb37e18b8b70" (UID: "afee2ffe-35e2-4903-849c-fb37e18b8b70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:54.287117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286976 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afee2ffe-35e2-4903-849c-fb37e18b8b70-istiod-ca-cert" pod "afee2ffe-35e2-4903-849c-fb37e18b8b70" (UID: "afee2ffe-35e2-4903-849c-fb37e18b8b70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:54.287117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.286988 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.287117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.287033 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.287117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.287051 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.287117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.287067 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.288546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.288523 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-podinfo" pod "afee2ffe-35e2-4903-849c-fb37e18b8b70" (UID: "afee2ffe-35e2-4903-849c-fb37e18b8b70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:54.288757 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.288734 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-token" pod "afee2ffe-35e2-4903-849c-fb37e18b8b70" (UID: "afee2ffe-35e2-4903-849c-fb37e18b8b70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:54.288874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.288840 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-kube-api-access-pmtfx" pod "afee2ffe-35e2-4903-849c-fb37e18b8b70" (UID: "afee2ffe-35e2-4903-849c-fb37e18b8b70"). InnerVolumeSpecName "kube-api-access-pmtfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:54.288874 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.288840 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-envoy" pod "afee2ffe-35e2-4903-849c-fb37e18b8b70" (UID: "afee2ffe-35e2-4903-849c-fb37e18b8b70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:54.387573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.387532 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmtfx\" (UniqueName: \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-kube-api-access-pmtfx\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.387573 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.387571 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afee2ffe-35e2-4903-849c-fb37e18b8b70-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.387824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.387587 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.387824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.387600 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.387824 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.387611 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afee2ffe-35e2-4903-849c-fb37e18b8b70-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.588195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.588097 4298 generic.go:356] "Generic (PLEG): container finished" podID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerID="4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb" exitCode=0 Nov 03 22:52:54.588195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.588150 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" Nov 03 22:52:54.588195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.588175 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" event={"ID":"afee2ffe-35e2-4903-849c-fb37e18b8b70","Type":"ContainerDied","Data":"4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb"} Nov 03 22:52:54.588466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.588253 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4" event={"ID":"afee2ffe-35e2-4903-849c-fb37e18b8b70","Type":"ContainerDied","Data":"d5e23fc61dd7f56fa2c8518e172b86da7f338269f4de97f3f50e8a8b3b009c96"} Nov 03 22:52:54.588466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.588275 4298 scope.go:122] "RemoveContainer" containerID="4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb" Nov 03 22:52:54.599253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.599233 4298 scope.go:122] "RemoveContainer" containerID="4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb" Nov 03 22:52:54.599553 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:52:54.599533 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb\": container with ID starting with 4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb not found: ID does not exist" containerID="4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb" Nov 03 22:52:54.599632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.599564 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb"} err="failed to get container status \"4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb\": rpc error: code = NotFound desc = could not find container \"4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb\": container with ID starting with 4f113f20ab838fb881539ff6e6a00cbf9bf1bbd07fb3fae4641ccd217afd47bb not found: ID does not exist" Nov 03 22:52:54.755564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.755528 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4"] Nov 03 22:52:54.760521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:54.760495 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-4xgx4"] Nov 03 22:52:55.001262 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:55.001232 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:55.138331 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:55.138300 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" path="/var/lib/kubelet/pods/afee2ffe-35e2-4903-849c-fb37e18b8b70/volumes" Nov 03 22:52:55.181099 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:55.181067 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:57.219584 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:52:57.219545 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:02.051089 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:02.051037 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:02.232956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:02.232922 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:03.471026 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:03.470904 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:04.142083 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:04.142039 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:04.327593 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:04.327564 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:06.364817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:06.364765 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:07.523313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:07.523278 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:07.896136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:07.896041 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:08.176286 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:08.176192 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:08.378524 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:08.378496 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s"] Nov 03 22:53:08.427781 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:08.427695 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f7e6db-9r46s"] Nov 03 22:53:08.741810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:08.741771 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:08.918967 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:08.918933 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:09.115134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:09.115052 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:09.136838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:09.136809 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9a9597-e202-4231-80f3-6c9cf7c915ca" path="/var/lib/kubelet/pods/5a9a9597-e202-4231-80f3-6c9cf7c915ca/volumes" Nov 03 22:53:09.401931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:09.401795 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:09.590014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:09.589985 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:09.700269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:09.700238 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7"] Nov 03 22:53:09.750016 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:09.749989 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-906673-nj2q7"] Nov 03 22:53:10.115135 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:10.115062 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:10.382177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:10.382095 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:10.648897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:10.648819 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:10.929546 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:10.929462 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7"] Nov 03 22:53:10.936102 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:10.936077 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:10.979800 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:10.979771 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2a6801-crxl7"] Nov 03 22:53:11.134037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:11.133998 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5210e1e3-1c7d-490d-be3f-541812789f02" path="/var/lib/kubelet/pods/5210e1e3-1c7d-490d-be3f-541812789f02/volumes" Nov 03 22:53:11.134500 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:11.134479 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af323b31-3c27-4a31-87e8-94a5ae96aafc" path="/var/lib/kubelet/pods/af323b31-3c27-4a31-87e8-94a5ae96aafc/volumes" Nov 03 22:53:11.353706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:11.353671 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:14.628773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:14.628734 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:14.830370 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:14.830340 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:14.878000 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:14.877959 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2"] Nov 03 22:53:14.878195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:14.878172 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" containerID="cri-o://d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829" gracePeriod=30 Nov 03 22:53:15.030922 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:15.030890 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:15.887574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:15.887541 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:16.105956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:16.105892 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:16.324840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:16.324807 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:17.169097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:17.169059 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:17.492851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:17.492816 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:17.565810 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:53:17.565769 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:53:17.600735 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:17.600702 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv"] Nov 03 22:53:17.600958 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:17.600936 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" containerID="cri-o://c5878756fed2b1db5cc77e9d8762e1c63a3793187085ee23cd53c9976bffe977" gracePeriod=30 Nov 03 22:53:17.673773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:17.673744 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:18.112427 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.112391 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:18.149306 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.149273 4298 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:53:18.149471 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.149326 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:53:18.551131 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551097 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn"] Nov 03 22:53:18.551523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551514 4298 cpu_manager.go:414] "containerMap: removing container" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerName="istio-proxy" Nov 03 22:53:18.551566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551525 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerName="istio-proxy" Nov 03 22:53:18.551566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551538 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" Nov 03 22:53:18.551566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551543 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" Nov 03 22:53:18.551566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551552 4298 cpu_manager.go:414] "containerMap: removing container" podUID="b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6" containerName="revoke-keys" Nov 03 22:53:18.551566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551557 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6" containerName="revoke-keys" Nov 03 22:53:18.551566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551567 4298 cpu_manager.go:414] "containerMap: removing container" podUID="5210e1e3-1c7d-490d-be3f-541812789f02" containerName="revoke-keys" Nov 03 22:53:18.551801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551572 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5210e1e3-1c7d-490d-be3f-541812789f02" containerName="revoke-keys" Nov 03 22:53:18.551801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551583 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1accc7f5-0255-43de-8ce1-5f892f8ff521" containerName="revoke-keys" Nov 03 22:53:18.551801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551588 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1accc7f5-0255-43de-8ce1-5f892f8ff521" containerName="revoke-keys" Nov 03 22:53:18.551801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551657 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="afee2ffe-35e2-4903-849c-fb37e18b8b70" containerName="istio-proxy" Nov 03 22:53:18.551801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551668 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" Nov 03 22:53:18.551801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551673 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6" containerName="revoke-keys" Nov 03 22:53:18.551801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551680 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="5210e1e3-1c7d-490d-be3f-541812789f02" containerName="revoke-keys" Nov 03 22:53:18.551801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551686 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1accc7f5-0255-43de-8ce1-5f892f8ff521" containerName="revoke-keys" Nov 03 22:53:18.551801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.551692 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" Nov 03 22:53:18.556188 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.556172 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:18.612689 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.612608 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn"] Nov 03 22:53:18.629396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.629362 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnfct\" (UniqueName: \"kubernetes.io/projected/efccac8d-69e4-4df5-9432-2c35585d9c5f-kube-api-access-wnfct\") pod \"maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:18.629578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.629405 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:18.629578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.629525 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:18.658971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.658943 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:53:18.659134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.659025 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:53:18.731133 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.731097 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:18.731348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.731197 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnfct\" (UniqueName: \"kubernetes.io/projected/efccac8d-69e4-4df5-9432-2c35585d9c5f-kube-api-access-wnfct\") pod \"maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:18.731348 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.731228 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:18.731854 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.731830 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:18.733547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.733527 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:18.787928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.787901 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnfct\" (UniqueName: \"kubernetes.io/projected/efccac8d-69e4-4df5-9432-2c35585d9c5f-kube-api-access-wnfct\") pod \"maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:18.869234 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.869145 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:18.958176 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:18.958144 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:19.233208 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:53:19.233177 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefccac8d_69e4_4df5_9432_2c35585d9c5f.slice/crio-9842abb647bf51da8c75e41d9f512a11e623412aed63d196a8d82992315b17d1 WatchSource:0}: Error finding container 9842abb647bf51da8c75e41d9f512a11e623412aed63d196a8d82992315b17d1: Status 404 returned error can't find the container with id 9842abb647bf51da8c75e41d9f512a11e623412aed63d196a8d82992315b17d1 Nov 03 22:53:19.237142 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:19.237120 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn"] Nov 03 22:53:19.736344 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:19.736298 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" event={"ID":"efccac8d-69e4-4df5-9432-2c35585d9c5f","Type":"ContainerStarted","Data":"caa52dba89173dfad9c00df80647617993c8e4337841ca67de2960ff5dca43cc"} Nov 03 22:53:19.736761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:19.736376 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" event={"ID":"efccac8d-69e4-4df5-9432-2c35585d9c5f","Type":"ContainerStarted","Data":"9842abb647bf51da8c75e41d9f512a11e623412aed63d196a8d82992315b17d1"} Nov 03 22:53:19.852028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:19.851973 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" podStartSLOduration=1.8519590639999999 podStartE2EDuration="1.851959064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:19.850229563 +0000 UTC m=+2756.830963216" watchObservedRunningTime="2026-11-03 22:53:19.851959064 +0000 UTC m=+2756.832692714" Nov 03 22:53:20.194796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.194771 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:53:20.270744 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.270662 4298 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:53:20.270744 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.270718 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:53:20.345560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.345524 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-data\") pod \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " Nov 03 22:53:20.345749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.345571 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-envoy\") pod \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " Nov 03 22:53:20.345749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.345604 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-token\" (UniqueName: \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-token\") pod \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " Nov 03 22:53:20.345749 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.345638 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-podinfo\") pod \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " Nov 03 22:53:20.345919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.345765 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istiod-ca-cert\") pod \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " Nov 03 22:53:20.345919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.345813 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-kube-api-access-6bwnn\" (UniqueName: \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-kube-api-access-6bwnn\") pod \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " Nov 03 22:53:20.345919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.345849 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-socket\") pod \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " Nov 03 22:53:20.345919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.345860 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-data" pod "d6cba416-b4fa-41a0-9a0f-9e3249d52c92" (UID: "d6cba416-b4fa-41a0-9a0f-9e3249d52c92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:20.345919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.345878 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-credential-socket\") pod \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " Nov 03 22:53:20.346213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.345933 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-certs\") pod \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\" (UID: \"d6cba416-b4fa-41a0-9a0f-9e3249d52c92\") " Nov 03 22:53:20.346213 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.346160 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-socket" pod "d6cba416-b4fa-41a0-9a0f-9e3249d52c92" (UID: "d6cba416-b4fa-41a0-9a0f-9e3249d52c92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:20.346338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.346215 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-credential-socket" pod "d6cba416-b4fa-41a0-9a0f-9e3249d52c92" (UID: "d6cba416-b4fa-41a0-9a0f-9e3249d52c92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:20.346400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.346339 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:20.346400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.346362 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:20.346400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.346378 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:20.346400 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.346387 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istiod-ca-cert" pod "d6cba416-b4fa-41a0-9a0f-9e3249d52c92" (UID: "d6cba416-b4fa-41a0-9a0f-9e3249d52c92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:20.346628 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.346396 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-certs" pod "d6cba416-b4fa-41a0-9a0f-9e3249d52c92" (UID: "d6cba416-b4fa-41a0-9a0f-9e3249d52c92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:20.348058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.348017 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-kube-api-access-6bwnn" pod "d6cba416-b4fa-41a0-9a0f-9e3249d52c92" (UID: "d6cba416-b4fa-41a0-9a0f-9e3249d52c92"). InnerVolumeSpecName "kube-api-access-6bwnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:20.348307 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.348282 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-token" pod "d6cba416-b4fa-41a0-9a0f-9e3249d52c92" (UID: "d6cba416-b4fa-41a0-9a0f-9e3249d52c92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:20.348426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.348324 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-envoy" pod "d6cba416-b4fa-41a0-9a0f-9e3249d52c92" (UID: "d6cba416-b4fa-41a0-9a0f-9e3249d52c92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:20.348426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.348337 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-podinfo" pod "d6cba416-b4fa-41a0-9a0f-9e3249d52c92" (UID: "d6cba416-b4fa-41a0-9a0f-9e3249d52c92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:20.447860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.447830 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:20.447860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.447856 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bwnn\" (UniqueName: \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-kube-api-access-6bwnn\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:20.447860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.447866 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:20.448105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.447875 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:20.448105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.447883 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:20.448105 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.447892 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cba416-b4fa-41a0-9a0f-9e3249d52c92-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:20.744161 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.744126 4298 generic.go:356] "Generic (PLEG): container finished" podID="efccac8d-69e4-4df5-9432-2c35585d9c5f" containerID="caa52dba89173dfad9c00df80647617993c8e4337841ca67de2960ff5dca43cc" exitCode=0 Nov 03 22:53:20.744574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.744215 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" event={"ID":"efccac8d-69e4-4df5-9432-2c35585d9c5f","Type":"ContainerDied","Data":"caa52dba89173dfad9c00df80647617993c8e4337841ca67de2960ff5dca43cc"} Nov 03 22:53:20.745839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.745816 4298 generic.go:356] "Generic (PLEG): container finished" podID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerID="d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829" exitCode=0 Nov 03 22:53:20.745908 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.745870 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" Nov 03 22:53:20.745943 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.745895 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" event={"ID":"d6cba416-b4fa-41a0-9a0f-9e3249d52c92","Type":"ContainerDied","Data":"d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829"} Nov 03 22:53:20.745981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.745947 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2" event={"ID":"d6cba416-b4fa-41a0-9a0f-9e3249d52c92","Type":"ContainerDied","Data":"0a01f26dd91dbb2674ac8a26050ed15fac51687ff95bf6f2e96d4f0038798a54"} Nov 03 22:53:20.745981 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.745970 4298 scope.go:122] "RemoveContainer" containerID="d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829" Nov 03 22:53:20.757221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.757203 4298 scope.go:122] "RemoveContainer" containerID="f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433" Nov 03 22:53:20.766832 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.766814 4298 scope.go:122] "RemoveContainer" containerID="d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829" Nov 03 22:53:20.767106 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:53:20.767086 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829\": container with ID starting with d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829 not found: ID does not exist" containerID="d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829" Nov 03 22:53:20.767175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.767116 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829"} err="failed to get container status \"d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829\": rpc error: code = NotFound desc = could not find container \"d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829\": container with ID starting with d754837cac1114c8fcedde17b40a7d27b690ab40b2b8baa308031a0c9e2a4829 not found: ID does not exist" Nov 03 22:53:20.767175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.767147 4298 scope.go:122] "RemoveContainer" containerID="f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433" Nov 03 22:53:20.767401 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:53:20.767383 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433\": container with ID starting with f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433 not found: ID does not exist" containerID="f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433" Nov 03 22:53:20.767456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.767408 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433"} err="failed to get container status \"f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433\": rpc error: code = NotFound desc = could not find container \"f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433\": container with ID starting with f2fc1a9984273ed42becaea35b942cc6a5b71de1def82cd39a1795aeb58cc433 not found: ID does not exist" Nov 03 22:53:20.970332 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:20.970293 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2"] Nov 03 22:53:21.020814 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:21.020732 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-77kn2"] Nov 03 22:53:21.133518 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:21.133487 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" path="/var/lib/kubelet/pods/d6cba416-b4fa-41a0-9a0f-9e3249d52c92/volumes" Nov 03 22:53:21.262366 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:21.262337 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:21.890478 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:21.890447 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:22.065204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.065126 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-xks-ca\") pod \"efccac8d-69e4-4df5-9432-2c35585d9c5f\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " Nov 03 22:53:22.065204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.065174 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efccac8d-69e4-4df5-9432-2c35585d9c5f-kube-api-access-wnfct\" (UniqueName: \"kubernetes.io/projected/efccac8d-69e4-4df5-9432-2c35585d9c5f-kube-api-access-wnfct\") pod \"efccac8d-69e4-4df5-9432-2c35585d9c5f\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " Nov 03 22:53:22.065424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.065212 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-openshift-ca\") pod \"efccac8d-69e4-4df5-9432-2c35585d9c5f\" (UID: \"efccac8d-69e4-4df5-9432-2c35585d9c5f\") " Nov 03 22:53:22.065706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.065674 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-openshift-ca" pod "efccac8d-69e4-4df5-9432-2c35585d9c5f" (UID: "efccac8d-69e4-4df5-9432-2c35585d9c5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:22.067306 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.067279 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-xks-ca" pod "efccac8d-69e4-4df5-9432-2c35585d9c5f" (UID: "efccac8d-69e4-4df5-9432-2c35585d9c5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:22.067404 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.067329 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efccac8d-69e4-4df5-9432-2c35585d9c5f-kube-api-access-wnfct" pod "efccac8d-69e4-4df5-9432-2c35585d9c5f" (UID: "efccac8d-69e4-4df5-9432-2c35585d9c5f"). InnerVolumeSpecName "kube-api-access-wnfct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:22.166169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.166136 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:22.166169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.166164 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnfct\" (UniqueName: \"kubernetes.io/projected/efccac8d-69e4-4df5-9432-2c35585d9c5f-kube-api-access-wnfct\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:22.166169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.166173 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efccac8d-69e4-4df5-9432-2c35585d9c5f-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:22.762696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.762655 4298 generic.go:356] "Generic (PLEG): container finished" podID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerID="c5878756fed2b1db5cc77e9d8762e1c63a3793187085ee23cd53c9976bffe977" exitCode=0 Nov 03 22:53:22.762885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.762673 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" event={"ID":"bad67035-dcf3-45d0-aad3-cca0ddd7c429","Type":"ContainerDied","Data":"c5878756fed2b1db5cc77e9d8762e1c63a3793187085ee23cd53c9976bffe977"} Nov 03 22:53:22.764263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.764243 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" Nov 03 22:53:22.764409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.764243 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn" event={"ID":"efccac8d-69e4-4df5-9432-2c35585d9c5f","Type":"ContainerDied","Data":"9842abb647bf51da8c75e41d9f512a11e623412aed63d196a8d82992315b17d1"} Nov 03 22:53:22.764409 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.764355 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9842abb647bf51da8c75e41d9f512a11e623412aed63d196a8d82992315b17d1" Nov 03 22:53:22.846565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.846540 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:53:22.973523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.973490 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-socket\") pod \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " Nov 03 22:53:22.974031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.973549 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-envoy\") pod \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " Nov 03 22:53:22.974031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.973653 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istiod-ca-cert\") pod \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " Nov 03 22:53:22.974031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.973693 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-podinfo\") pod \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " Nov 03 22:53:22.974031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.973736 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-data\") pod \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " Nov 03 22:53:22.974031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.973831 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-credential-socket\") pod \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " Nov 03 22:53:22.974031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.973849 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-socket" pod "bad67035-dcf3-45d0-aad3-cca0ddd7c429" (UID: "bad67035-dcf3-45d0-aad3-cca0ddd7c429"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:22.974031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.973863 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-kube-api-access-4znf2\" (UniqueName: \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-kube-api-access-4znf2\") pod \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " Nov 03 22:53:22.974031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.973906 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-token\" (UniqueName: \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-token\") pod \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " Nov 03 22:53:22.974031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.973978 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-certs\") pod \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\" (UID: \"bad67035-dcf3-45d0-aad3-cca0ddd7c429\") " Nov 03 22:53:22.974511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.974062 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-data" pod "bad67035-dcf3-45d0-aad3-cca0ddd7c429" (UID: "bad67035-dcf3-45d0-aad3-cca0ddd7c429"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:22.974511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.974095 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-credential-socket" pod "bad67035-dcf3-45d0-aad3-cca0ddd7c429" (UID: "bad67035-dcf3-45d0-aad3-cca0ddd7c429"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:22.974511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.974366 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-certs" pod "bad67035-dcf3-45d0-aad3-cca0ddd7c429" (UID: "bad67035-dcf3-45d0-aad3-cca0ddd7c429"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:22.974511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.974424 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:22.974511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.974446 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:22.974511 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.974460 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:22.974780 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.974681 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istiod-ca-cert" pod "bad67035-dcf3-45d0-aad3-cca0ddd7c429" (UID: "bad67035-dcf3-45d0-aad3-cca0ddd7c429"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:22.976216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.976190 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-kube-api-access-4znf2" pod "bad67035-dcf3-45d0-aad3-cca0ddd7c429" (UID: "bad67035-dcf3-45d0-aad3-cca0ddd7c429"). InnerVolumeSpecName "kube-api-access-4znf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:22.976349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.976271 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-envoy" pod "bad67035-dcf3-45d0-aad3-cca0ddd7c429" (UID: "bad67035-dcf3-45d0-aad3-cca0ddd7c429"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:22.976349 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.976310 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-token" pod "bad67035-dcf3-45d0-aad3-cca0ddd7c429" (UID: "bad67035-dcf3-45d0-aad3-cca0ddd7c429"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:22.976844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:22.976828 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-podinfo" pod "bad67035-dcf3-45d0-aad3-cca0ddd7c429" (UID: "bad67035-dcf3-45d0-aad3-cca0ddd7c429"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:23.033692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.033632 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km"] Nov 03 22:53:23.039236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.039208 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf"] Nov 03 22:53:23.042292 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.042263 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp"] Nov 03 22:53:23.050985 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.050957 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp"] Nov 03 22:53:23.075693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.075666 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4znf2\" (UniqueName: \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-kube-api-access-4znf2\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.075693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.075693 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.075876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.075705 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.075876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.075713 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.075876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.075722 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.075876 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.075730 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad67035-dcf3-45d0-aad3-cca0ddd7c429-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.224173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.224087 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:23.573256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.573165 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:23.770741 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.770710 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" event={"ID":"bad67035-dcf3-45d0-aad3-cca0ddd7c429","Type":"ContainerDied","Data":"81e3e4a6f214545241e2c3a0990cfe214786cce191373c10168c0dbe6ee9292e"} Nov 03 22:53:23.770926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.770766 4298 scope.go:122] "RemoveContainer" containerID="c5878756fed2b1db5cc77e9d8762e1c63a3793187085ee23cd53c9976bffe977" Nov 03 22:53:23.770926 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.770769 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv" Nov 03 22:53:23.937864 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.937778 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv"] Nov 03 22:53:23.987744 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:23.987715 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-scwrv"] Nov 03 22:53:25.134498 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:25.134465 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" path="/var/lib/kubelet/pods/bad67035-dcf3-45d0-aad3-cca0ddd7c429/volumes" Nov 03 22:53:26.423963 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:26.423929 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:29.058074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:29.058043 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:30.468822 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:30.468788 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:30.702074 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:30.702038 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:31.237624 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:31.237586 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:33.313246 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:33.313212 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:33.498265 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:33.498234 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:35.473251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473220 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v"] Nov 03 22:53:35.473687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473576 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" Nov 03 22:53:35.473687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473587 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef2fc6c-a501-4a62-adcf-879a017e7bd8" containerName="istio-proxy" Nov 03 22:53:35.473687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473610 4298 cpu_manager.go:414] "containerMap: removing container" podUID="efccac8d-69e4-4df5-9432-2c35585d9c5f" containerName="revoke-keys" Nov 03 22:53:35.473687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473615 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efccac8d-69e4-4df5-9432-2c35585d9c5f" containerName="revoke-keys" Nov 03 22:53:35.473687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473629 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" Nov 03 22:53:35.473687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473635 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" Nov 03 22:53:35.473687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473663 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" Nov 03 22:53:35.473687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473669 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" Nov 03 22:53:35.473687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473679 4298 cpu_manager.go:414] "containerMap: removing container" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" Nov 03 22:53:35.473687 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473684 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" Nov 03 22:53:35.474025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473737 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad67035-dcf3-45d0-aad3-cca0ddd7c429" containerName="istio-proxy" Nov 03 22:53:35.474025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473750 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="efccac8d-69e4-4df5-9432-2c35585d9c5f" containerName="revoke-keys" Nov 03 22:53:35.474025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473757 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" Nov 03 22:53:35.474025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.473765 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6cba416-b4fa-41a0-9a0f-9e3249d52c92" containerName="istio-proxy" Nov 03 22:53:35.476940 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.476925 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:35.494717 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.494692 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d19310-mgj8v\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:35.494838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.494742 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbc4m\" (UniqueName: \"kubernetes.io/projected/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-kube-api-access-rbc4m\") pod \"maas-api-revoke-sub-72d19310-mgj8v\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:35.494888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.494868 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d19310-mgj8v\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:35.532692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.532665 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:53:35.537965 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.537933 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v"] Nov 03 22:53:35.583056 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.583024 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:53:35.595594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.595565 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d19310-mgj8v\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:35.595772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.595621 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d19310-mgj8v\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:35.595772 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.595673 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbc4m\" (UniqueName: \"kubernetes.io/projected/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-kube-api-access-rbc4m\") pod \"maas-api-revoke-sub-72d19310-mgj8v\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:35.596230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.596210 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d19310-mgj8v\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:35.598077 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.598056 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d19310-mgj8v\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:35.652753 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.652724 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbc4m\" (UniqueName: \"kubernetes.io/projected/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-kube-api-access-rbc4m\") pod \"maas-api-revoke-sub-72d19310-mgj8v\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:35.789709 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:35.789609 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:36.156973 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:53:36.156935 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46a7ddb_9ca7_4b6f_bc1b_43d00109d759.slice/crio-314e9ffdafed8fae056144dba00eef6398f3254c328ac76fea43ee46821e09a3 WatchSource:0}: Error finding container 314e9ffdafed8fae056144dba00eef6398f3254c328ac76fea43ee46821e09a3: Status 404 returned error can't find the container with id 314e9ffdafed8fae056144dba00eef6398f3254c328ac76fea43ee46821e09a3 Nov 03 22:53:36.158487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:36.158464 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v"] Nov 03 22:53:36.848417 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:36.848379 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" event={"ID":"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759","Type":"ContainerStarted","Data":"2c64cf755e1d07c2f9415f6c485e50087fc9d0fd0b2eaf87cfc5a5fa83c053d2"} Nov 03 22:53:36.848865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:36.848428 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" event={"ID":"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759","Type":"ContainerStarted","Data":"314e9ffdafed8fae056144dba00eef6398f3254c328ac76fea43ee46821e09a3"} Nov 03 22:53:36.965005 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:36.964950 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" podStartSLOduration=1.9649336179999999 podStartE2EDuration="1.964933618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:36.961075574 +0000 UTC m=+2773.941809227" watchObservedRunningTime="2026-11-03 22:53:36.964933618 +0000 UTC m=+2773.945667273" Nov 03 22:53:37.855078 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:37.855044 4298 generic.go:356] "Generic (PLEG): container finished" podID="d46a7ddb-9ca7-4b6f-bc1b-43d00109d759" containerID="2c64cf755e1d07c2f9415f6c485e50087fc9d0fd0b2eaf87cfc5a5fa83c053d2" exitCode=0 Nov 03 22:53:37.855560 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:37.855126 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" event={"ID":"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759","Type":"ContainerDied","Data":"2c64cf755e1d07c2f9415f6c485e50087fc9d0fd0b2eaf87cfc5a5fa83c053d2"} Nov 03 22:53:38.056690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.056654 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" containerID="cri-o://0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8" gracePeriod=45 Nov 03 22:53:38.060549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.060515 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" containerID="cri-o://7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5" gracePeriod=45 Nov 03 22:53:38.061493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.061238 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" containerID="cri-o://05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610" gracePeriod=45 Nov 03 22:53:38.077079 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.077048 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" containerID="cri-o://a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2" gracePeriod=45 Nov 03 22:53:38.169662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.169613 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused" Nov 03 22:53:38.337068 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.337042 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:53:38.357979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.357954 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:53:38.361674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.361632 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:53:38.366894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.366872 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:53:38.422768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.422685 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-model-cache\") pod \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " Nov 03 22:53:38.422768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.422724 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-model-cache\") pod \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " Nov 03 22:53:38.422768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.422743 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026f1bf5-559b-492e-a9f4-a433644d790d-tls-certs\" (UniqueName: \"kubernetes.io/secret/026f1bf5-559b-492e-a9f4-a433644d790d-tls-certs\") pod \"026f1bf5-559b-492e-a9f4-a433644d790d\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " Nov 03 22:53:38.422768 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.422764 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tls-certs\" (UniqueName: \"kubernetes.io/secret/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tls-certs\") pod \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.422787 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894f31d0-02aa-445d-9ba8-8628e8d41e0e-kube-api-access-nz4lt\" (UniqueName: \"kubernetes.io/projected/894f31d0-02aa-445d-9ba8-8628e8d41e0e-kube-api-access-nz4lt\") pod \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.422813 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-model-cache\") pod \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.422831 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tmp-dir\") pod \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.422850 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-dshm\") pod \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.422888 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tmp-dir\") pod \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.422930 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tls-certs\") pod \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.422980 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-kube-api-access-pztmg\" (UniqueName: \"kubernetes.io/projected/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-kube-api-access-pztmg\") pod \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423033 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-dshm\") pod \"026f1bf5-559b-492e-a9f4-a433644d790d\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423059 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-model-cache" pod "894f31d0-02aa-445d-9ba8-8628e8d41e0e" (UID: "894f31d0-02aa-445d-9ba8-8628e8d41e0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423078 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-model-cache\") pod \"026f1bf5-559b-492e-a9f4-a433644d790d\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423080 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-model-cache" pod "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" (UID: "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.423124 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423110 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-dshm\") pod \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423147 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-tmp-dir\") pod \"026f1bf5-559b-492e-a9f4-a433644d790d\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423171 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-home\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-home\") pod \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423211 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-home\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-home\") pod \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423240 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026f1bf5-559b-492e-a9f4-a433644d790d-kube-api-access-lf5w9\" (UniqueName: \"kubernetes.io/projected/026f1bf5-559b-492e-a9f4-a433644d790d-kube-api-access-lf5w9\") pod \"026f1bf5-559b-492e-a9f4-a433644d790d\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423265 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c4c1d8-cb12-43fe-b9fe-861e31095b13-kube-api-access-4gbnh\" (UniqueName: \"kubernetes.io/projected/25c4c1d8-cb12-43fe-b9fe-861e31095b13-kube-api-access-4gbnh\") pod \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423289 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-home\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-home\") pod \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\" (UID: \"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f\") " Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423317 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tls-certs\") pod \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\" (UID: \"894f31d0-02aa-445d-9ba8-8628e8d41e0e\") " Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423342 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tmp-dir\") pod \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423368 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-home\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-home\") pod \"026f1bf5-559b-492e-a9f4-a433644d790d\" (UID: \"026f1bf5-559b-492e-a9f4-a433644d790d\") " Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423404 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-dshm\") pod \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\" (UID: \"25c4c1d8-cb12-43fe-b9fe-861e31095b13\") " Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423780 4298 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-model-cache\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.423830 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423800 4298 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-model-cache\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.426300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423204 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tmp-dir" pod "894f31d0-02aa-445d-9ba8-8628e8d41e0e" (UID: "894f31d0-02aa-445d-9ba8-8628e8d41e0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.426300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.426258 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-dshm" pod "894f31d0-02aa-445d-9ba8-8628e8d41e0e" (UID: "894f31d0-02aa-445d-9ba8-8628e8d41e0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.426300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423380 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tmp-dir" pod "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" (UID: "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.426300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423545 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-home" pod "25c4c1d8-cb12-43fe-b9fe-861e31095b13" (UID: "25c4c1d8-cb12-43fe-b9fe-861e31095b13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.426300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.423586 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-model-cache" pod "25c4c1d8-cb12-43fe-b9fe-861e31095b13" (UID: "25c4c1d8-cb12-43fe-b9fe-861e31095b13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.426300 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.426240 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-dshm" pod "25c4c1d8-cb12-43fe-b9fe-861e31095b13" (UID: "25c4c1d8-cb12-43fe-b9fe-861e31095b13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.426773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.426310 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894f31d0-02aa-445d-9ba8-8628e8d41e0e-kube-api-access-nz4lt" pod "894f31d0-02aa-445d-9ba8-8628e8d41e0e" (UID: "894f31d0-02aa-445d-9ba8-8628e8d41e0e"). InnerVolumeSpecName "kube-api-access-nz4lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:38.426773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.426494 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026f1bf5-559b-492e-a9f4-a433644d790d-tls-certs" pod "026f1bf5-559b-492e-a9f4-a433644d790d" (UID: "026f1bf5-559b-492e-a9f4-a433644d790d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:38.426773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.426589 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-tmp-dir" pod "026f1bf5-559b-492e-a9f4-a433644d790d" (UID: "026f1bf5-559b-492e-a9f4-a433644d790d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.426773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.426735 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-home" pod "894f31d0-02aa-445d-9ba8-8628e8d41e0e" (UID: "894f31d0-02aa-445d-9ba8-8628e8d41e0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.427048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.426826 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tls-certs" pod "25c4c1d8-cb12-43fe-b9fe-861e31095b13" (UID: "25c4c1d8-cb12-43fe-b9fe-861e31095b13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:38.427048 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.427027 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-dshm" pod "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" (UID: "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.427177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.427054 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-home" pod "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" (UID: "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.427177 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.427080 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tls-certs" pod "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" (UID: "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:38.427376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.427292 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-model-cache" pod "026f1bf5-559b-492e-a9f4-a433644d790d" (UID: "026f1bf5-559b-492e-a9f4-a433644d790d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.427376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.427307 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-kube-api-access-pztmg" pod "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" (UID: "3cd70cf3-1d97-4c3a-9ada-bed7517ca13f"). InnerVolumeSpecName "kube-api-access-pztmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:38.427564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.427319 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-home" pod "026f1bf5-559b-492e-a9f4-a433644d790d" (UID: "026f1bf5-559b-492e-a9f4-a433644d790d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.427564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.427362 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tmp-dir" pod "25c4c1d8-cb12-43fe-b9fe-861e31095b13" (UID: "25c4c1d8-cb12-43fe-b9fe-861e31095b13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.428451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.428432 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-dshm" pod "026f1bf5-559b-492e-a9f4-a433644d790d" (UID: "026f1bf5-559b-492e-a9f4-a433644d790d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:38.428620 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.428604 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026f1bf5-559b-492e-a9f4-a433644d790d-kube-api-access-lf5w9" pod "026f1bf5-559b-492e-a9f4-a433644d790d" (UID: "026f1bf5-559b-492e-a9f4-a433644d790d"). InnerVolumeSpecName "kube-api-access-lf5w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:38.428724 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.428710 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c4c1d8-cb12-43fe-b9fe-861e31095b13-kube-api-access-4gbnh" pod "25c4c1d8-cb12-43fe-b9fe-861e31095b13" (UID: "25c4c1d8-cb12-43fe-b9fe-861e31095b13"). InnerVolumeSpecName "kube-api-access-4gbnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:38.428829 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.428813 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tls-certs" pod "894f31d0-02aa-445d-9ba8-8628e8d41e0e" (UID: "894f31d0-02aa-445d-9ba8-8628e8d41e0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:38.525137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525100 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pztmg\" (UniqueName: \"kubernetes.io/projected/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-kube-api-access-pztmg\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525137 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525139 4298 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-dshm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525154 4298 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-model-cache\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525168 4298 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-dshm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525183 4298 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-tmp-dir\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525195 4298 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-home\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525207 4298 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-home\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525219 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf5w9\" (UniqueName: \"kubernetes.io/projected/026f1bf5-559b-492e-a9f4-a433644d790d-kube-api-access-lf5w9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525235 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gbnh\" (UniqueName: \"kubernetes.io/projected/25c4c1d8-cb12-43fe-b9fe-861e31095b13-kube-api-access-4gbnh\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525248 4298 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-home\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525261 4298 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tls-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525273 4298 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tmp-dir\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525286 4298 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026f1bf5-559b-492e-a9f4-a433644d790d-home\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525298 4298 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-dshm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525312 4298 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c4c1d8-cb12-43fe-b9fe-861e31095b13-model-cache\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525325 4298 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026f1bf5-559b-492e-a9f4-a433644d790d-tls-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525342 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525338 4298 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c4c1d8-cb12-43fe-b9fe-861e31095b13-tls-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525351 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz4lt\" (UniqueName: \"kubernetes.io/projected/894f31d0-02aa-445d-9ba8-8628e8d41e0e-kube-api-access-nz4lt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525367 4298 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894f31d0-02aa-445d-9ba8-8628e8d41e0e-tmp-dir\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525381 4298 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-dshm\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525392 4298 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tmp-dir\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.525838 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.525404 4298 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f-tls-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:38.863185 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.863153 4298 generic.go:356] "Generic (PLEG): container finished" podID="026f1bf5-559b-492e-a9f4-a433644d790d" containerID="a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2" exitCode=0 Nov 03 22:53:38.863692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.863217 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" Nov 03 22:53:38.863692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.863230 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" event={"ID":"026f1bf5-559b-492e-a9f4-a433644d790d","Type":"ContainerDied","Data":"a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2"} Nov 03 22:53:38.863692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.863281 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp" event={"ID":"026f1bf5-559b-492e-a9f4-a433644d790d","Type":"ContainerDied","Data":"62c9323ef45fa45aa71280c91b924568b8327219a051288e6f4cbaf372d91b5e"} Nov 03 22:53:38.863692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.863314 4298 scope.go:122] "RemoveContainer" containerID="a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2" Nov 03 22:53:38.864836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.864809 4298 generic.go:356] "Generic (PLEG): container finished" podID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerID="7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5" exitCode=0 Nov 03 22:53:38.864979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.864864 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" Nov 03 22:53:38.864979 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.864906 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" event={"ID":"894f31d0-02aa-445d-9ba8-8628e8d41e0e","Type":"ContainerDied","Data":"7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5"} Nov 03 22:53:38.865103 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.865000 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf" event={"ID":"894f31d0-02aa-445d-9ba8-8628e8d41e0e","Type":"ContainerDied","Data":"2cd79a2cd2548796cba6ff7a27b9642d7449ef3c1a006e6c5837e24e3f2cafd6"} Nov 03 22:53:38.866424 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.866400 4298 generic.go:356] "Generic (PLEG): container finished" podID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerID="0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8" exitCode=0 Nov 03 22:53:38.866601 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.866585 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" event={"ID":"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f","Type":"ContainerDied","Data":"0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8"} Nov 03 22:53:38.866696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.866611 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" event={"ID":"3cd70cf3-1d97-4c3a-9ada-bed7517ca13f","Type":"ContainerDied","Data":"b42e80b24f064ec59af298795308a30cbd44057ad24838dbe03c8640df5c6a3c"} Nov 03 22:53:38.866696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.866680 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km" Nov 03 22:53:38.872143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.872121 4298 generic.go:356] "Generic (PLEG): container finished" podID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerID="05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610" exitCode=0 Nov 03 22:53:38.872386 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.872183 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" Nov 03 22:53:38.872490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.872214 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" event={"ID":"25c4c1d8-cb12-43fe-b9fe-861e31095b13","Type":"ContainerDied","Data":"05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610"} Nov 03 22:53:38.872490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.872423 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp" event={"ID":"25c4c1d8-cb12-43fe-b9fe-861e31095b13","Type":"ContainerDied","Data":"9ffa1214c854d9f1f4c6a2b9e15ca870efbe050f1b2cc6004e3f1e7a1ae2e217"} Nov 03 22:53:38.876905 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.876880 4298 scope.go:122] "RemoveContainer" containerID="a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2" Nov 03 22:53:38.877212 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:53:38.877187 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2\": container with ID starting with a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2 not found: ID does not exist" containerID="a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2" Nov 03 22:53:38.877297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.877225 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2"} err="failed to get container status \"a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2\": rpc error: code = NotFound desc = could not find container \"a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2\": container with ID starting with a7f534b3dc3839e968d3c8073ef51264cbe991742b2a551cc11bed6878ce39f2 not found: ID does not exist" Nov 03 22:53:38.877297 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.877248 4298 scope.go:122] "RemoveContainer" containerID="7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5" Nov 03 22:53:38.886930 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.886910 4298 scope.go:122] "RemoveContainer" containerID="7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5" Nov 03 22:53:38.887221 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:53:38.887202 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5\": container with ID starting with 7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5 not found: ID does not exist" containerID="7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5" Nov 03 22:53:38.887294 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.887230 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5"} err="failed to get container status \"7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5\": rpc error: code = NotFound desc = could not find container \"7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5\": container with ID starting with 7d6197ac88078e93fa769ae9808d61f40bcd040a19a21d26390f2521471729f5 not found: ID does not exist" Nov 03 22:53:38.887294 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.887253 4298 scope.go:122] "RemoveContainer" containerID="0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8" Nov 03 22:53:38.895462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.895446 4298 scope.go:122] "RemoveContainer" containerID="0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8" Nov 03 22:53:38.895773 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:53:38.895755 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8\": container with ID starting with 0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8 not found: ID does not exist" containerID="0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8" Nov 03 22:53:38.895831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.895780 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8"} err="failed to get container status \"0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8\": rpc error: code = NotFound desc = could not find container \"0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8\": container with ID starting with 0f32e7f41eff36840c268854cb517bd74fce2ccdf122ced078145a509c289ae8 not found: ID does not exist" Nov 03 22:53:38.895831 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.895796 4298 scope.go:122] "RemoveContainer" containerID="05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610" Nov 03 22:53:38.904473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.904453 4298 scope.go:122] "RemoveContainer" containerID="05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610" Nov 03 22:53:38.904887 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:53:38.904863 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610\": container with ID starting with 05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610 not found: ID does not exist" containerID="05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610" Nov 03 22:53:38.904982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.904895 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610"} err="failed to get container status \"05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610\": rpc error: code = NotFound desc = could not find container \"05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610\": container with ID starting with 05c6a94b2addad66cc8b0fcea904b861be2be04082dceb1024c53d3741a08610 not found: ID does not exist" Nov 03 22:53:38.995369 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:38.995344 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:39.029533 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.029509 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km"] Nov 03 22:53:39.029891 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.029867 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-xks-ca\") pod \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " Nov 03 22:53:39.029944 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.029913 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-openshift-ca\") pod \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " Nov 03 22:53:39.029992 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.029950 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-kube-api-access-rbc4m\" (UniqueName: \"kubernetes.io/projected/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-kube-api-access-rbc4m\") pod \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\" (UID: \"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759\") " Nov 03 22:53:39.030197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.030175 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-openshift-ca" pod "d46a7ddb-9ca7-4b6f-bc1b-43d00109d759" (UID: "d46a7ddb-9ca7-4b6f-bc1b-43d00109d759"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:39.031927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.031900 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-xks-ca" pod "d46a7ddb-9ca7-4b6f-bc1b-43d00109d759" (UID: "d46a7ddb-9ca7-4b6f-bc1b-43d00109d759"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:39.031927 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.031920 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-kube-api-access-rbc4m" pod "d46a7ddb-9ca7-4b6f-bc1b-43d00109d759" (UID: "d46a7ddb-9ca7-4b6f-bc1b-43d00109d759"). InnerVolumeSpecName "kube-api-access-rbc4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:39.032606 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.032586 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-2-w5-f81f8f-kserve-69bf799d56-g66km"] Nov 03 22:53:39.130911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.130838 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:39.130911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.130861 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:39.130911 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.130872 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbc4m\" (UniqueName: \"kubernetes.io/projected/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759-kube-api-access-rbc4m\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:39.133387 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.133363 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" path="/var/lib/kubelet/pods/3cd70cf3-1d97-4c3a-9ada-bed7517ca13f/volumes" Nov 03 22:53:39.140443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.140420 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf"] Nov 03 22:53:39.143296 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.143275 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/distinct-w5-f81f8f-kserve-66c55565cf-7b8qf"] Nov 03 22:53:39.250440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.250401 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp"] Nov 03 22:53:39.256223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.256200 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/simulator-w5-f81f8f-kserve-c9dc6c57c-2hgdp"] Nov 03 22:53:39.365663 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.365616 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp"] Nov 03 22:53:39.368669 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.368634 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f81f8f/premium-w5-f81f8f-kserve-7c6ccbff5b-fgjgp"] Nov 03 22:53:39.881730 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.881701 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" Nov 03 22:53:39.882198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.881694 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v" event={"ID":"d46a7ddb-9ca7-4b6f-bc1b-43d00109d759","Type":"ContainerDied","Data":"314e9ffdafed8fae056144dba00eef6398f3254c328ac76fea43ee46821e09a3"} Nov 03 22:53:39.882198 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:39.881828 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="314e9ffdafed8fae056144dba00eef6398f3254c328ac76fea43ee46821e09a3" Nov 03 22:53:41.133320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:41.133290 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" path="/var/lib/kubelet/pods/026f1bf5-559b-492e-a9f4-a433644d790d/volumes" Nov 03 22:53:41.133827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:41.133696 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" path="/var/lib/kubelet/pods/25c4c1d8-cb12-43fe-b9fe-861e31095b13/volumes" Nov 03 22:53:41.134051 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:41.134037 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" path="/var/lib/kubelet/pods/894f31d0-02aa-445d-9ba8-8628e8d41e0e/volumes" Nov 03 22:53:47.551818 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:53:47.551780 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:53:51.005846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:51.005811 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:52.573582 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:52.573548 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:52.830118 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:52.830029 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:53.007338 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:53.007304 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn"] Nov 03 22:53:53.057309 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:53.057268 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f81f8f-8rgsn"] Nov 03 22:53:53.133778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:53.133702 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efccac8d-69e4-4df5-9432-2c35585d9c5f" path="/var/lib/kubelet/pods/efccac8d-69e4-4df5-9432-2c35585d9c5f/volumes" Nov 03 22:53:53.475527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:53:53.475496 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:00.271165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:00.271126 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:00.464901 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:00.464863 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:00.606795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:00.606717 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj"] Nov 03 22:54:00.606960 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:00.606937 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" containerID="cri-o://b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b" gracePeriod=30 Nov 03 22:54:00.657520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:00.657489 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:01.098502 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:01.098469 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:05.852770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.852746 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:54:05.979197 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979167 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-podinfo\") pod \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " Nov 03 22:54:05.979396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979231 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istiod-ca-cert\") pod \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " Nov 03 22:54:05.979396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979261 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-certs\") pod \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " Nov 03 22:54:05.979396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979283 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-credential-socket\") pod \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " Nov 03 22:54:05.979396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979312 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-data\") pod \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " Nov 03 22:54:05.979396 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979346 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-envoy\") pod \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " Nov 03 22:54:05.979739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979439 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-token\") pod \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " Nov 03 22:54:05.979739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979485 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-kube-api-access-pxgbt\" (UniqueName: \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-kube-api-access-pxgbt\") pod \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " Nov 03 22:54:05.979739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979526 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-socket\") pod \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\" (UID: \"b4cfe907-f90c-4555-86a4-5eee9c7dcc00\") " Nov 03 22:54:05.979739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979630 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-certs" pod "b4cfe907-f90c-4555-86a4-5eee9c7dcc00" (UID: "b4cfe907-f90c-4555-86a4-5eee9c7dcc00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:05.979739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979692 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-credential-socket" pod "b4cfe907-f90c-4555-86a4-5eee9c7dcc00" (UID: "b4cfe907-f90c-4555-86a4-5eee9c7dcc00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:05.979739 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979704 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-data" pod "b4cfe907-f90c-4555-86a4-5eee9c7dcc00" (UID: "b4cfe907-f90c-4555-86a4-5eee9c7dcc00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:05.980061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979753 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istiod-ca-cert" pod "b4cfe907-f90c-4555-86a4-5eee9c7dcc00" (UID: "b4cfe907-f90c-4555-86a4-5eee9c7dcc00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:05.980061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979876 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-socket" pod "b4cfe907-f90c-4555-86a4-5eee9c7dcc00" (UID: "b4cfe907-f90c-4555-86a4-5eee9c7dcc00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:05.980061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979953 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.980061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979974 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.980061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.979989 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.980061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.980004 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.981686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.981664 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-podinfo" pod "b4cfe907-f90c-4555-86a4-5eee9c7dcc00" (UID: "b4cfe907-f90c-4555-86a4-5eee9c7dcc00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:54:05.981686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.981664 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-token" pod "b4cfe907-f90c-4555-86a4-5eee9c7dcc00" (UID: "b4cfe907-f90c-4555-86a4-5eee9c7dcc00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:05.981844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.981719 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-kube-api-access-pxgbt" pod "b4cfe907-f90c-4555-86a4-5eee9c7dcc00" (UID: "b4cfe907-f90c-4555-86a4-5eee9c7dcc00"). InnerVolumeSpecName "kube-api-access-pxgbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:05.981844 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:05.981774 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-envoy" pod "b4cfe907-f90c-4555-86a4-5eee9c7dcc00" (UID: "b4cfe907-f90c-4555-86a4-5eee9c7dcc00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:06.034139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.033259 4298 generic.go:356] "Generic (PLEG): container finished" podID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerID="b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b" exitCode=0 Nov 03 22:54:06.034139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.033387 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" event={"ID":"b4cfe907-f90c-4555-86a4-5eee9c7dcc00","Type":"ContainerDied","Data":"b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b"} Nov 03 22:54:06.034139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.033478 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" event={"ID":"b4cfe907-f90c-4555-86a4-5eee9c7dcc00","Type":"ContainerDied","Data":"d48d54f5868b5f5b2576fd82f568ab5ddb61f1e73f3fdd1b0648aa0be09af0a6"} Nov 03 22:54:06.034139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.033514 4298 scope.go:122] "RemoveContainer" containerID="b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b" Nov 03 22:54:06.034139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.033741 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj" Nov 03 22:54:06.045268 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.045248 4298 scope.go:122] "RemoveContainer" containerID="2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec" Nov 03 22:54:06.054025 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.054007 4298 scope.go:122] "RemoveContainer" containerID="b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b" Nov 03 22:54:06.054296 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:54:06.054277 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b\": container with ID starting with b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b not found: ID does not exist" containerID="b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b" Nov 03 22:54:06.054344 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.054305 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b"} err="failed to get container status \"b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b\": rpc error: code = NotFound desc = could not find container \"b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b\": container with ID starting with b4462a4872aa097011e859e28bdec7ef6c3c773df8b9ecd16a1d8b2a59e4b02b not found: ID does not exist" Nov 03 22:54:06.054344 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.054323 4298 scope.go:122] "RemoveContainer" containerID="2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec" Nov 03 22:54:06.054625 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:54:06.054607 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec\": container with ID starting with 2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec not found: ID does not exist" containerID="2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec" Nov 03 22:54:06.054683 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.054633 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec"} err="failed to get container status \"2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec\": rpc error: code = NotFound desc = could not find container \"2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec\": container with ID starting with 2571008431ced1402057b2d12d8dd160d3c0bf86a9d9a6aba425307b741939ec not found: ID does not exist" Nov 03 22:54:06.080859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.080826 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:06.081006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.080863 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:06.081006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.080881 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxgbt\" (UniqueName: \"kubernetes.io/projected/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-kube-api-access-pxgbt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:06.081006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.080897 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:06.081006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.080912 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfe907-f90c-4555-86a4-5eee9c7dcc00-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:06.201916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.201880 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj"] Nov 03 22:54:06.204976 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.204946 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-h8zbj"] Nov 03 22:54:06.748430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:06.748397 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:07.134041 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:07.133969 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" path="/var/lib/kubelet/pods/b4cfe907-f90c-4555-86a4-5eee9c7dcc00/volumes" Nov 03 22:54:17.551832 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:54:17.551786 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:54:19.300872 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:19.300840 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:23.914856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:23.914820 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:27.398166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:27.398136 4298 scope.go:122] "RemoveContainer" containerID="f2aafd3c0a98c4636e4752cdb18f78b7511609d974f0b23be1bdc02a6e905e89" Nov 03 22:54:27.406721 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:27.406703 4298 scope.go:122] "RemoveContainer" containerID="4dd0c6d6d097d37729707e8fe66d316a73e96cb2c1affbf1e5272ff9e95f9d94" Nov 03 22:54:29.395344 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.395306 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:29.513019 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.512985 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd"] Nov 03 22:54:29.513367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513355 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" Nov 03 22:54:29.513411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513368 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" Nov 03 22:54:29.513411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513387 4298 cpu_manager.go:414] "containerMap: removing container" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" Nov 03 22:54:29.513411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513392 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" Nov 03 22:54:29.513411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513402 4298 cpu_manager.go:414] "containerMap: removing container" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" Nov 03 22:54:29.513411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513407 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513414 4298 cpu_manager.go:414] "containerMap: removing container" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513419 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513430 4298 cpu_manager.go:414] "containerMap: removing container" podUID="d46a7ddb-9ca7-4b6f-bc1b-43d00109d759" containerName="revoke-keys" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513435 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46a7ddb-9ca7-4b6f-bc1b-43d00109d759" containerName="revoke-keys" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513445 4298 cpu_manager.go:414] "containerMap: removing container" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513449 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513457 4298 cpu_manager.go:414] "containerMap: removing container" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513464 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513512 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46a7ddb-9ca7-4b6f-bc1b-43d00109d759" containerName="revoke-keys" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513522 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="894f31d0-02aa-445d-9ba8-8628e8d41e0e" containerName="main" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513528 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c4c1d8-cb12-43fe-b9fe-861e31095b13" containerName="main" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513534 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd70cf3-1d97-4c3a-9ada-bed7517ca13f" containerName="main" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513541 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="026f1bf5-559b-492e-a9f4-a433644d790d" containerName="main" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513547 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" Nov 03 22:54:29.513577 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.513554 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4cfe907-f90c-4555-86a4-5eee9c7dcc00" containerName="istio-proxy" Nov 03 22:54:29.518070 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.518052 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:29.571032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.571007 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:54:29.575139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.575108 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd"] Nov 03 22:54:29.580602 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.580576 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-25xcd\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:29.580723 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.580620 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lbv\" (UniqueName: \"kubernetes.io/projected/3042e29b-04a1-4ab2-adaf-20e0b3630395-kube-api-access-d6lbv\") pod \"maas-api-revoke-sub-85237e45-25xcd\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:29.580777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.580754 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-25xcd\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:29.621346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.621320 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:54:29.681278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.681211 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lbv\" (UniqueName: \"kubernetes.io/projected/3042e29b-04a1-4ab2-adaf-20e0b3630395-kube-api-access-d6lbv\") pod \"maas-api-revoke-sub-85237e45-25xcd\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:29.681278 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.681258 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-25xcd\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:29.681474 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.681307 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-25xcd\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:29.681972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.681953 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-25xcd\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:29.683895 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.683865 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-25xcd\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:29.737792 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.737767 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lbv\" (UniqueName: \"kubernetes.io/projected/3042e29b-04a1-4ab2-adaf-20e0b3630395-kube-api-access-d6lbv\") pod \"maas-api-revoke-sub-85237e45-25xcd\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:29.831111 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:29.831073 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:30.196975 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:54:30.196941 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3042e29b_04a1_4ab2_adaf_20e0b3630395.slice/crio-6672f534e6e9bcca528b3a02f7542e7175445318e00c64ac0fb59214ada50c09 WatchSource:0}: Error finding container 6672f534e6e9bcca528b3a02f7542e7175445318e00c64ac0fb59214ada50c09: Status 404 returned error can't find the container with id 6672f534e6e9bcca528b3a02f7542e7175445318e00c64ac0fb59214ada50c09 Nov 03 22:54:30.200180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:30.200158 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd"] Nov 03 22:54:30.592493 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:30.592461 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:30.765451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:30.765419 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:31.169277 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:31.169244 4298 generic.go:356] "Generic (PLEG): container finished" podID="3042e29b-04a1-4ab2-adaf-20e0b3630395" containerID="5c1878043e57bf8923b78bc279706e1f6674c70dc637cb78185529f3b555ea90" exitCode=0 Nov 03 22:54:31.169469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:31.169333 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" event={"ID":"3042e29b-04a1-4ab2-adaf-20e0b3630395","Type":"ContainerDied","Data":"5c1878043e57bf8923b78bc279706e1f6674c70dc637cb78185529f3b555ea90"} Nov 03 22:54:31.169469 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:31.169394 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" event={"ID":"3042e29b-04a1-4ab2-adaf-20e0b3630395","Type":"ContainerStarted","Data":"6672f534e6e9bcca528b3a02f7542e7175445318e00c64ac0fb59214ada50c09"} Nov 03 22:54:32.302123 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:32.302098 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:32.407816 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:32.407782 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-xks-ca\") pod \"3042e29b-04a1-4ab2-adaf-20e0b3630395\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " Nov 03 22:54:32.407990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:32.407847 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-openshift-ca\") pod \"3042e29b-04a1-4ab2-adaf-20e0b3630395\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " Nov 03 22:54:32.407990 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:32.407888 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3042e29b-04a1-4ab2-adaf-20e0b3630395-kube-api-access-d6lbv\" (UniqueName: \"kubernetes.io/projected/3042e29b-04a1-4ab2-adaf-20e0b3630395-kube-api-access-d6lbv\") pod \"3042e29b-04a1-4ab2-adaf-20e0b3630395\" (UID: \"3042e29b-04a1-4ab2-adaf-20e0b3630395\") " Nov 03 22:54:32.408221 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:32.408195 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-openshift-ca" pod "3042e29b-04a1-4ab2-adaf-20e0b3630395" (UID: "3042e29b-04a1-4ab2-adaf-20e0b3630395"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:32.410006 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:32.409986 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-xks-ca" pod "3042e29b-04a1-4ab2-adaf-20e0b3630395" (UID: "3042e29b-04a1-4ab2-adaf-20e0b3630395"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:32.410065 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:32.410003 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3042e29b-04a1-4ab2-adaf-20e0b3630395-kube-api-access-d6lbv" pod "3042e29b-04a1-4ab2-adaf-20e0b3630395" (UID: "3042e29b-04a1-4ab2-adaf-20e0b3630395"). InnerVolumeSpecName "kube-api-access-d6lbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:32.509251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:32.509214 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6lbv\" (UniqueName: \"kubernetes.io/projected/3042e29b-04a1-4ab2-adaf-20e0b3630395-kube-api-access-d6lbv\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:32.509251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:32.509245 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:32.509251 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:32.509256 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3042e29b-04a1-4ab2-adaf-20e0b3630395-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:33.180804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:33.180773 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" Nov 03 22:54:33.180804 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:33.180780 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd" event={"ID":"3042e29b-04a1-4ab2-adaf-20e0b3630395","Type":"ContainerDied","Data":"6672f534e6e9bcca528b3a02f7542e7175445318e00c64ac0fb59214ada50c09"} Nov 03 22:54:33.181011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:33.180835 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6672f534e6e9bcca528b3a02f7542e7175445318e00c64ac0fb59214ada50c09" Nov 03 22:54:41.881353 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:41.881318 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:43.972126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:54:43.972090 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:47.550847 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:54:47.550808 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:55:17.551034 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:55:17.550996 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:55:19.682220 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:19.682185 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:24.262129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:24.262091 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:25.047312 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:25.047272 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf"] Nov 03 22:55:25.097773 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:25.097745 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4196cd7a-6pmbf"] Nov 03 22:55:25.134425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:25.134394 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4962b98b-981e-4950-8b9e-7d147fd40b81" path="/var/lib/kubelet/pods/4962b98b-981e-4950-8b9e-7d147fd40b81/volumes" Nov 03 22:55:26.661092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:26.661060 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:27.452460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:27.452428 4298 scope.go:122] "RemoveContainer" containerID="d0ae1c8d48bb4ae2beead722e9e27a74985486a1a9ad897e98c47f728818ee9e" Nov 03 22:55:27.461507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:27.461488 4298 scope.go:122] "RemoveContainer" containerID="5d4af59fcfe9dbad460d7cadbc0f9694d02d0c41699dfc62a417007712b22310" Nov 03 22:55:27.469616 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:27.469595 4298 scope.go:122] "RemoveContainer" containerID="4f6112509fb7df646aa7697be0faeb5eb42e46cf5884fc27d2ea0ede6ebe6021" Nov 03 22:55:29.804449 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:29.804415 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:35.763931 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.763896 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb"] Nov 03 22:55:35.764451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.764278 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3042e29b-04a1-4ab2-adaf-20e0b3630395" containerName="revoke-keys" Nov 03 22:55:35.764451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.764289 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3042e29b-04a1-4ab2-adaf-20e0b3630395" containerName="revoke-keys" Nov 03 22:55:35.764451 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.764379 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3042e29b-04a1-4ab2-adaf-20e0b3630395" containerName="revoke-keys" Nov 03 22:55:35.769007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.768990 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:35.830503 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.830471 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:55:35.873549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.873515 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:55:35.876402 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.876381 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb"] Nov 03 22:55:35.895840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.895810 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hklqb\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:35.896012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.895879 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hklqb\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:35.896012 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.895977 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vlt\" (UniqueName: \"kubernetes.io/projected/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-kube-api-access-z6vlt\") pod \"maas-api-revoke-sub-c1e271a9-hklqb\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:35.996850 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.996810 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vlt\" (UniqueName: \"kubernetes.io/projected/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-kube-api-access-z6vlt\") pod \"maas-api-revoke-sub-c1e271a9-hklqb\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:35.997027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.996886 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hklqb\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:35.997027 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.996956 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hklqb\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:35.997662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.997616 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hklqb\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:35.999257 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:35.999229 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hklqb\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:36.055140 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:36.055071 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vlt\" (UniqueName: \"kubernetes.io/projected/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-kube-api-access-z6vlt\") pod \"maas-api-revoke-sub-c1e271a9-hklqb\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:36.081207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:36.081172 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:36.448206 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:55:36.448168 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc54ea82_a25c_41d7_b635_f86ed4f7d0fd.slice/crio-e7543284960571f83d40fc57d1945468b007b97b6a3948fb5bdfc776538f7bb7 WatchSource:0}: Error finding container e7543284960571f83d40fc57d1945468b007b97b6a3948fb5bdfc776538f7bb7: Status 404 returned error can't find the container with id e7543284960571f83d40fc57d1945468b007b97b6a3948fb5bdfc776538f7bb7 Nov 03 22:55:36.450230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:36.450203 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb"] Nov 03 22:55:36.505339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:36.505305 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" event={"ID":"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd","Type":"ContainerStarted","Data":"e7543284960571f83d40fc57d1945468b007b97b6a3948fb5bdfc776538f7bb7"} Nov 03 22:55:37.512440 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:37.512401 4298 generic.go:356] "Generic (PLEG): container finished" podID="dc54ea82-a25c-41d7-b635-f86ed4f7d0fd" containerID="26a4f062ddc31b4ea86efbf62a11e310e51161110c9069c7e126dbaf702e3304" exitCode=0 Nov 03 22:55:37.512902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:37.512492 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" event={"ID":"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd","Type":"ContainerDied","Data":"26a4f062ddc31b4ea86efbf62a11e310e51161110c9069c7e126dbaf702e3304"} Nov 03 22:55:38.052024 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.051995 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7"] Nov 03 22:55:38.054632 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.054612 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e3466bb-t2kj7"] Nov 03 22:55:38.646547 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.646521 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:38.824011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.823920 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-openshift-ca\") pod \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " Nov 03 22:55:38.824011 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.824008 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-xks-ca\") pod \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " Nov 03 22:55:38.824322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.824048 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-kube-api-access-z6vlt\" (UniqueName: \"kubernetes.io/projected/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-kube-api-access-z6vlt\") pod \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\" (UID: \"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd\") " Nov 03 22:55:38.824322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.824278 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-openshift-ca" pod "dc54ea82-a25c-41d7-b635-f86ed4f7d0fd" (UID: "dc54ea82-a25c-41d7-b635-f86ed4f7d0fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:38.824464 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.824385 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:38.826185 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.826150 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-kube-api-access-z6vlt" pod "dc54ea82-a25c-41d7-b635-f86ed4f7d0fd" (UID: "dc54ea82-a25c-41d7-b635-f86ed4f7d0fd"). InnerVolumeSpecName "kube-api-access-z6vlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:38.826185 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.826152 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-xks-ca" pod "dc54ea82-a25c-41d7-b635-f86ed4f7d0fd" (UID: "dc54ea82-a25c-41d7-b635-f86ed4f7d0fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:55:38.925777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.925743 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:38.925777 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:38.925774 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6vlt\" (UniqueName: \"kubernetes.io/projected/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd-kube-api-access-z6vlt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:39.133308 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:39.133224 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6caa58b-8191-4ac1-b316-1e3ec7b4724f" path="/var/lib/kubelet/pods/f6caa58b-8191-4ac1-b316-1e3ec7b4724f/volumes" Nov 03 22:55:39.524706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:39.524668 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" event={"ID":"dc54ea82-a25c-41d7-b635-f86ed4f7d0fd","Type":"ContainerDied","Data":"e7543284960571f83d40fc57d1945468b007b97b6a3948fb5bdfc776538f7bb7"} Nov 03 22:55:39.524706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:39.524687 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb" Nov 03 22:55:39.524918 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:39.524725 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7543284960571f83d40fc57d1945468b007b97b6a3948fb5bdfc776538f7bb7" Nov 03 22:55:44.723186 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:44.723150 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:44.913363 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:44.913336 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:47.552846 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:55:47.552812 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:55:54.626388 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.626349 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq"] Nov 03 22:55:54.626795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.626707 4298 cpu_manager.go:414] "containerMap: removing container" podUID="dc54ea82-a25c-41d7-b635-f86ed4f7d0fd" containerName="revoke-keys" Nov 03 22:55:54.626795 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.626718 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc54ea82-a25c-41d7-b635-f86ed4f7d0fd" containerName="revoke-keys" Nov 03 22:55:54.626881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.626800 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc54ea82-a25c-41d7-b635-f86ed4f7d0fd" containerName="revoke-keys" Nov 03 22:55:54.629836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.629820 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.668007 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.667976 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.668149 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.668013 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmn4\" (UniqueName: \"kubernetes.io/projected/818a85a1-2bd0-4d9e-84ef-951f647247e2-kube-api-access-8gmn4\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.668149 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.668043 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.668236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.668148 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.668236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.668167 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.668236 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.668194 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818a85a1-2bd0-4d9e-84ef-951f647247e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.668435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.668332 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.668435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.668387 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.668435 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.668418 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.676851 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.676829 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6"] Nov 03 22:55:54.680187 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.680173 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.690507 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.690485 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kf4rv\"" Nov 03 22:55:54.734531 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.734494 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq"] Nov 03 22:55:54.769762 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.769736 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.769919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.769776 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.769919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.769805 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701f8c70-a62b-487e-8008-04258f4d75b2-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.769919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.769871 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzhj\" (UniqueName: \"kubernetes.io/projected/701f8c70-a62b-487e-8008-04258f4d75b2-kube-api-access-rtzhj\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.769919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.769900 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.769919 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.769918 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.770217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.769936 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.770217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.769993 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.770217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770055 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.770217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770086 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmn4\" (UniqueName: \"kubernetes.io/projected/818a85a1-2bd0-4d9e-84ef-951f647247e2-kube-api-access-8gmn4\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.770217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770138 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.770217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770170 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.770217 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770192 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.770596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770239 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818a85a1-2bd0-4d9e-84ef-951f647247e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.770596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770266 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.770596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770238 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.770596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770272 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.770596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770329 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.770596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770352 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.770596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770359 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f8c70-a62b-487e-8008-04258f4d75b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.770596 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770466 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f8c70-a62b-487e-8008-04258f4d75b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.770973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770620 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.770973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.770878 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818a85a1-2bd0-4d9e-84ef-951f647247e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.772625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.772601 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.772891 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.772869 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.788426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.788397 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6"] Nov 03 22:55:54.828350 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.828320 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818a85a1-2bd0-4d9e-84ef-951f647247e2-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.828563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.828540 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmn4\" (UniqueName: \"kubernetes.io/projected/818a85a1-2bd0-4d9e-84ef-951f647247e2-kube-api-access-8gmn4\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6x4mq\" (UID: \"818a85a1-2bd0-4d9e-84ef-951f647247e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.871578 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.871550 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.871759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.871584 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.871759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.871599 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f8c70-a62b-487e-8008-04258f4d75b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.871759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.871634 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f8c70-a62b-487e-8008-04258f4d75b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.871759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.871666 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.871759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.871687 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701f8c70-a62b-487e-8008-04258f4d75b2-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.871759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.871705 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzhj\" (UniqueName: \"kubernetes.io/projected/701f8c70-a62b-487e-8008-04258f4d75b2-kube-api-access-rtzhj\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.871759 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.871741 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.872136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.871908 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.872136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.872031 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.872260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.872135 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.872319 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.872278 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.872393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.872348 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.872393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.872379 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f8c70-a62b-487e-8008-04258f4d75b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.873975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.873951 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f8c70-a62b-487e-8008-04258f4d75b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.874233 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.874214 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f8c70-a62b-487e-8008-04258f4d75b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.932603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.932538 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzhj\" (UniqueName: \"kubernetes.io/projected/701f8c70-a62b-487e-8008-04258f4d75b2-kube-api-access-rtzhj\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.932820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.932800 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701f8c70-a62b-487e-8008-04258f4d75b2-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-tqdf6\" (UID: \"701f8c70-a62b-487e-8008-04258f4d75b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:54.941862 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.941843 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:54.992994 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:54.992964 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:55.361959 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:55:55.361930 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818a85a1_2bd0_4d9e_84ef_951f647247e2.slice/crio-ee2c35f32694133e0f6690787594e7ac3c66d5a91c1b655e634cafdbd65b4f01 WatchSource:0}: Error finding container ee2c35f32694133e0f6690787594e7ac3c66d5a91c1b655e634cafdbd65b4f01: Status 404 returned error can't find the container with id ee2c35f32694133e0f6690787594e7ac3c66d5a91c1b655e634cafdbd65b4f01 Nov 03 22:55:55.364304 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.364273 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:55:55.364390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.364337 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:55:55.364390 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.364374 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:55:55.365572 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.365551 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq"] Nov 03 22:55:55.411386 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:55:55.411358 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701f8c70_a62b_487e_8008_04258f4d75b2.slice/crio-816391c9ec0fcc04ba2fc17f73bd1b010319bd664e6f81c08d4b77ba4bb6494b WatchSource:0}: Error finding container 816391c9ec0fcc04ba2fc17f73bd1b010319bd664e6f81c08d4b77ba4bb6494b: Status 404 returned error can't find the container with id 816391c9ec0fcc04ba2fc17f73bd1b010319bd664e6f81c08d4b77ba4bb6494b Nov 03 22:55:55.413885 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.413850 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:55:55.413966 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.413953 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:55:55.414028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.413987 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:55:55.414250 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.414231 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6"] Nov 03 22:55:55.609058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.608969 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" event={"ID":"818a85a1-2bd0-4d9e-84ef-951f647247e2","Type":"ContainerStarted","Data":"aa0dc269665662778e874867e0125cc21567e8b0fc468e0074135d33aa8ba5b3"} Nov 03 22:55:55.609058 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.609031 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" event={"ID":"818a85a1-2bd0-4d9e-84ef-951f647247e2","Type":"ContainerStarted","Data":"ee2c35f32694133e0f6690787594e7ac3c66d5a91c1b655e634cafdbd65b4f01"} Nov 03 22:55:55.610535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.610513 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" event={"ID":"701f8c70-a62b-487e-8008-04258f4d75b2","Type":"ContainerStarted","Data":"76fe9588a5e0ea90017006587efefb5df87be5770a1dae3515e1995f76146b39"} Nov 03 22:55:55.610675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.610549 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" event={"ID":"701f8c70-a62b-487e-8008-04258f4d75b2","Type":"ContainerStarted","Data":"816391c9ec0fcc04ba2fc17f73bd1b010319bd664e6f81c08d4b77ba4bb6494b"} Nov 03 22:55:55.730812 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.730751 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" podStartSLOduration=1.730736461 podStartE2EDuration="1.730736461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:55.72686014 +0000 UTC m=+2912.707593793" watchObservedRunningTime="2026-11-03 22:55:55.730736461 +0000 UTC m=+2912.711470191" Nov 03 22:55:55.856891 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.856823 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" podStartSLOduration=1.856799792 podStartE2EDuration="1.856799792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:55.852151105 +0000 UTC m=+2912.832884774" watchObservedRunningTime="2026-11-03 22:55:55.856799792 +0000 UTC m=+2912.837533446" Nov 03 22:55:55.944732 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.943730 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:55:55.993339 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:55.993310 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:55:56.946413 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:56.946372 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 03 22:55:56.947076 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:56.947045 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" podUID="818a85a1-2bd0-4d9e-84ef-951f647247e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 03 22:55:56.995311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:56.995280 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 22:55:56.995551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:56.995529 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" podUID="701f8c70-a62b-487e-8008-04258f4d75b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 22:55:57.976095 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:57.976057 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 03 22:55:57.976745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:57.976715 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" podUID="818a85a1-2bd0-4d9e-84ef-951f647247e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 03 22:55:57.993323 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:57.993297 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 22:55:57.993523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:57.993504 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" podUID="701f8c70-a62b-487e-8008-04258f4d75b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 22:55:58.945670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:58.945061 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 03 22:55:58.945670 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:58.945137 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" podUID="818a85a1-2bd0-4d9e-84ef-951f647247e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 03 22:55:58.993872 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:58.993838 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 22:55:58.994269 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:58.993894 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" podUID="701f8c70-a62b-487e-8008-04258f4d75b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 22:55:59.946677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:59.945024 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 03 22:55:59.946677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:59.945087 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" podUID="818a85a1-2bd0-4d9e-84ef-951f647247e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 03 22:55:59.994003 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:59.993967 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 22:55:59.994410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:55:59.994031 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" podUID="701f8c70-a62b-487e-8008-04258f4d75b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 22:56:00.945093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:00.945051 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 03 22:56:00.945399 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:00.945368 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" podUID="818a85a1-2bd0-4d9e-84ef-951f647247e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 03 22:56:00.993758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:00.993722 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 22:56:00.993938 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:00.993786 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" podUID="701f8c70-a62b-487e-8008-04258f4d75b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 22:56:01.942363 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:01.942321 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 03 22:56:01.942817 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:01.942395 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" podUID="818a85a1-2bd0-4d9e-84ef-951f647247e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 03 22:56:01.994001 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:01.993964 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 22:56:01.994143 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:01.994034 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" podUID="701f8c70-a62b-487e-8008-04258f4d75b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 22:56:03.713737 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:03.713638 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:56:03.714256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:03.714235 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:56:03.715032 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:03.715000 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" Nov 03 22:56:03.726496 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:03.726465 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:56:03.726836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:03.726812 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:56:03.727743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:03.727726 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" Nov 03 22:56:07.275570 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.275521 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6"] Nov 03 22:56:07.279256 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.279234 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:07.333310 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.333280 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:56:07.337806 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.337773 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6"] Nov 03 22:56:07.383758 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.383725 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:56:07.427959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.427912 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgtjw\" (UniqueName: \"kubernetes.io/projected/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-kube-api-access-zgtjw\") pod \"maas-api-revoke-sub-2e1f5b5f-ckcg6\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:07.427959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.427971 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ckcg6\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:07.428169 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.428080 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ckcg6\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:07.528625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.528531 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ckcg6\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:07.528839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.528632 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgtjw\" (UniqueName: \"kubernetes.io/projected/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-kube-api-access-zgtjw\") pod \"maas-api-revoke-sub-2e1f5b5f-ckcg6\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:07.528839 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.528700 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ckcg6\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:07.529322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.529296 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ckcg6\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:07.530865 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.530838 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ckcg6\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:07.585499 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.585465 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgtjw\" (UniqueName: \"kubernetes.io/projected/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-kube-api-access-zgtjw\") pod \"maas-api-revoke-sub-2e1f5b5f-ckcg6\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:07.592289 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.592266 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:07.956169 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:56:07.956137 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7618db_7a07_4634_8ddc_fa2f5f0390d5.slice/crio-4647e0f7914fb45595bb013b58356c8e82967825850bd2f81d96c74c93f50bfd WatchSource:0}: Error finding container 4647e0f7914fb45595bb013b58356c8e82967825850bd2f81d96c74c93f50bfd: Status 404 returned error can't find the container with id 4647e0f7914fb45595bb013b58356c8e82967825850bd2f81d96c74c93f50bfd Nov 03 22:56:07.960239 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:07.960216 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6"] Nov 03 22:56:08.724430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:08.724396 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" event={"ID":"6a7618db-7a07-4634-8ddc-fa2f5f0390d5","Type":"ContainerStarted","Data":"db44d14c5165231a66fa1cab5100f033b0871336e4cd6f184ce699f89df000ab"} Nov 03 22:56:08.724941 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:08.724451 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" event={"ID":"6a7618db-7a07-4634-8ddc-fa2f5f0390d5","Type":"ContainerStarted","Data":"4647e0f7914fb45595bb013b58356c8e82967825850bd2f81d96c74c93f50bfd"} Nov 03 22:56:08.841807 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:08.841751 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" podStartSLOduration=1.841737065 podStartE2EDuration="1.841737065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:08.839536168 +0000 UTC m=+2925.820269820" watchObservedRunningTime="2026-11-03 22:56:08.841737065 +0000 UTC m=+2925.822470717" Nov 03 22:56:09.048178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.048074 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj"] Nov 03 22:56:09.050972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.050947 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3010b426-rgnjj"] Nov 03 22:56:09.133719 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.133690 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d77a26-67b3-4553-bacd-bc88287d4672" path="/var/lib/kubelet/pods/27d77a26-67b3-4553-bacd-bc88287d4672/volumes" Nov 03 22:56:09.731175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.731136 4298 generic.go:356] "Generic (PLEG): container finished" podID="6a7618db-7a07-4634-8ddc-fa2f5f0390d5" containerID="db44d14c5165231a66fa1cab5100f033b0871336e4cd6f184ce699f89df000ab" exitCode=0 Nov 03 22:56:09.731597 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.731224 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" event={"ID":"6a7618db-7a07-4634-8ddc-fa2f5f0390d5","Type":"ContainerDied","Data":"db44d14c5165231a66fa1cab5100f033b0871336e4cd6f184ce699f89df000ab"} Nov 03 22:56:09.735170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.735146 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7"] Nov 03 22:56:09.738555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.738541 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.845428 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.845394 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7"] Nov 03 22:56:09.849057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.849032 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn9nw\" (UniqueName: \"kubernetes.io/projected/da4f3451-f4d3-408e-aad0-67f89b6f7260-kube-api-access-qn9nw\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.849230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.849067 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.849230 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.849092 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.849380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.849230 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.849380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.849279 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.849380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.849305 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.849380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.849332 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.849586 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.849394 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.849655 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.849602 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4f3451-f4d3-408e-aad0-67f89b6f7260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950320 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950291 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4f3451-f4d3-408e-aad0-67f89b6f7260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950335 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn9nw\" (UniqueName: \"kubernetes.io/projected/da4f3451-f4d3-408e-aad0-67f89b6f7260-kube-api-access-qn9nw\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950358 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950381 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950418 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950441 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950456 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950505 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950483 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950537 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950883 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950907 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.950972 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.950949 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.951157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.951078 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.951199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.951162 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4f3451-f4d3-408e-aad0-67f89b6f7260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.952834 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.952805 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:09.952974 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:09.952954 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:10.007604 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.007521 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4f3451-f4d3-408e-aad0-67f89b6f7260-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:10.007787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.007689 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn9nw\" (UniqueName: \"kubernetes.io/projected/da4f3451-f4d3-408e-aad0-67f89b6f7260-kube-api-access-qn9nw\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-6rjg7\" (UID: \"da4f3451-f4d3-408e-aad0-67f89b6f7260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:10.047193 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.047158 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8"] Nov 03 22:56:10.050322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.050297 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4be605-v6mn8"] Nov 03 22:56:10.050465 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.050455 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:10.464591 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:56:10.464545 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4f3451_f4d3_408e_aad0_67f89b6f7260.slice/crio-2e77e581be8ed995eee7248abaa1efb4e4f7357f831c8e7a4ca039c4c6e62ef3 WatchSource:0}: Error finding container 2e77e581be8ed995eee7248abaa1efb4e4f7357f831c8e7a4ca039c4c6e62ef3: Status 404 returned error can't find the container with id 2e77e581be8ed995eee7248abaa1efb4e4f7357f831c8e7a4ca039c4c6e62ef3 Nov 03 22:56:10.466871 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.466835 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:56:10.466939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.466921 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:56:10.466982 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.466962 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 22:56:10.467833 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.467810 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7"] Nov 03 22:56:10.740585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.740488 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" event={"ID":"da4f3451-f4d3-408e-aad0-67f89b6f7260","Type":"ContainerStarted","Data":"c35a5a29024c8c3a0daf81d19a0eaad597b8316a4fa6b4657064451caaa350c7"} Nov 03 22:56:10.740585 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.740541 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" event={"ID":"da4f3451-f4d3-408e-aad0-67f89b6f7260","Type":"ContainerStarted","Data":"2e77e581be8ed995eee7248abaa1efb4e4f7357f831c8e7a4ca039c4c6e62ef3"} Nov 03 22:56:10.858731 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.858683 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" podStartSLOduration=1.858628789 podStartE2EDuration="1.858628789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:10.858271633 +0000 UTC m=+2927.839005286" watchObservedRunningTime="2026-11-03 22:56:10.858628789 +0000 UTC m=+2927.839362442" Nov 03 22:56:10.864462 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.864438 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:10.961166 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.961132 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-kube-api-access-zgtjw\" (UniqueName: \"kubernetes.io/projected/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-kube-api-access-zgtjw\") pod \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " Nov 03 22:56:10.961307 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.961238 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-xks-ca\") pod \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " Nov 03 22:56:10.961307 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.961291 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-openshift-ca\") pod \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\" (UID: \"6a7618db-7a07-4634-8ddc-fa2f5f0390d5\") " Nov 03 22:56:10.961899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.961872 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-openshift-ca" pod "6a7618db-7a07-4634-8ddc-fa2f5f0390d5" (UID: "6a7618db-7a07-4634-8ddc-fa2f5f0390d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:10.963527 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.963503 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-xks-ca" pod "6a7618db-7a07-4634-8ddc-fa2f5f0390d5" (UID: "6a7618db-7a07-4634-8ddc-fa2f5f0390d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:10.963757 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:10.963733 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-kube-api-access-zgtjw" pod "6a7618db-7a07-4634-8ddc-fa2f5f0390d5" (UID: "6a7618db-7a07-4634-8ddc-fa2f5f0390d5"). InnerVolumeSpecName "kube-api-access-zgtjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:11.051354 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:11.051266 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:11.052856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:11.052830 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 22:56:11.053004 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:11.052886 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" podUID="da4f3451-f4d3-408e-aad0-67f89b6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 22:56:11.062343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:11.062320 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgtjw\" (UniqueName: \"kubernetes.io/projected/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-kube-api-access-zgtjw\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:11.062343 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:11.062343 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:11.062494 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:11.062353 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7618db-7a07-4634-8ddc-fa2f5f0390d5-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:11.152778 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:11.152744 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4" path="/var/lib/kubelet/pods/01c1b4fe-4e0c-4b1e-95d1-3c8e96b0b3b4/volumes" Nov 03 22:56:11.754267 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:11.754236 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" Nov 03 22:56:11.754835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:11.754273 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6" event={"ID":"6a7618db-7a07-4634-8ddc-fa2f5f0390d5","Type":"ContainerDied","Data":"4647e0f7914fb45595bb013b58356c8e82967825850bd2f81d96c74c93f50bfd"} Nov 03 22:56:11.754835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:11.754330 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4647e0f7914fb45595bb013b58356c8e82967825850bd2f81d96c74c93f50bfd" Nov 03 22:56:12.051175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:12.051089 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 22:56:12.051175 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:12.051150 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" podUID="da4f3451-f4d3-408e-aad0-67f89b6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 22:56:13.051554 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:13.051516 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 22:56:13.051987 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:13.051569 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" podUID="da4f3451-f4d3-408e-aad0-67f89b6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 22:56:14.050910 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:14.050867 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 22:56:14.051157 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:14.050933 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" podUID="da4f3451-f4d3-408e-aad0-67f89b6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 22:56:15.051763 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:15.051729 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 22:56:15.052180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:15.051805 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" podUID="da4f3451-f4d3-408e-aad0-67f89b6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 22:56:16.051189 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:16.051153 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 22:56:16.051360 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:16.051221 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" podUID="da4f3451-f4d3-408e-aad0-67f89b6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 22:56:17.051625 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:17.051583 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 22:56:17.052064 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:17.051667 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" podUID="da4f3451-f4d3-408e-aad0-67f89b6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 22:56:17.552325 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:56:17.552284 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:56:18.701353 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:18.701313 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:18.701894 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:18.701678 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:18.702475 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:18.702456 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" Nov 03 22:56:19.775291 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.775258 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z"] Nov 03 22:56:19.775724 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.775707 4298 cpu_manager.go:414] "containerMap: removing container" podUID="6a7618db-7a07-4634-8ddc-fa2f5f0390d5" containerName="revoke-keys" Nov 03 22:56:19.775776 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.775726 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7618db-7a07-4634-8ddc-fa2f5f0390d5" containerName="revoke-keys" Nov 03 22:56:19.775813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.775804 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7618db-7a07-4634-8ddc-fa2f5f0390d5" containerName="revoke-keys" Nov 03 22:56:19.779952 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.779936 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:19.883563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.883522 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:56:19.883563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.883563 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:56:19.886677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.886656 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z"] Nov 03 22:56:19.944887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.944855 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-fb26z\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:19.945071 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.944920 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-fb26z\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:19.945150 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.945108 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pm6m\" (UniqueName: \"kubernetes.io/projected/dbb213fc-764a-4f5b-9866-043e798f6a4c-kube-api-access-8pm6m\") pod \"maas-api-revoke-sub-ec843018-fb26z\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:19.987076 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:19.987046 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:20.045983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:20.045894 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pm6m\" (UniqueName: \"kubernetes.io/projected/dbb213fc-764a-4f5b-9866-043e798f6a4c-kube-api-access-8pm6m\") pod \"maas-api-revoke-sub-ec843018-fb26z\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:20.045983 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:20.045982 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-fb26z\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:20.046199 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:20.046016 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-fb26z\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:20.046667 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:20.046633 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-fb26z\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:20.048311 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:20.048290 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-fb26z\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:20.106232 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:20.106197 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pm6m\" (UniqueName: \"kubernetes.io/projected/dbb213fc-764a-4f5b-9866-043e798f6a4c-kube-api-access-8pm6m\") pod \"maas-api-revoke-sub-ec843018-fb26z\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:20.392589 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:20.392478 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:20.806238 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:20.806205 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z"] Nov 03 22:56:21.812472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:21.812440 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" event={"ID":"dbb213fc-764a-4f5b-9866-043e798f6a4c","Type":"ContainerStarted","Data":"ef28b8208db484b277ea8749f243241aa36cd45128b4c38d2320e3c76955c8aa"} Nov 03 22:56:21.812810 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:21.812504 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" event={"ID":"dbb213fc-764a-4f5b-9866-043e798f6a4c","Type":"ContainerStarted","Data":"1793f78dbbbc510a028a1ca713a78539593c4039515973003f3eac352454c1ab"} Nov 03 22:56:22.819802 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:22.819770 4298 generic.go:356] "Generic (PLEG): container finished" podID="dbb213fc-764a-4f5b-9866-043e798f6a4c" containerID="ef28b8208db484b277ea8749f243241aa36cd45128b4c38d2320e3c76955c8aa" exitCode=0 Nov 03 22:56:22.820247 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:22.819858 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" event={"ID":"dbb213fc-764a-4f5b-9866-043e798f6a4c","Type":"ContainerDied","Data":"ef28b8208db484b277ea8749f243241aa36cd45128b4c38d2320e3c76955c8aa"} Nov 03 22:56:23.126973 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.126948 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:23.199975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.199947 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-xks-ca\") pod \"dbb213fc-764a-4f5b-9866-043e798f6a4c\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " Nov 03 22:56:23.200151 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.200007 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb213fc-764a-4f5b-9866-043e798f6a4c-kube-api-access-8pm6m\" (UniqueName: \"kubernetes.io/projected/dbb213fc-764a-4f5b-9866-043e798f6a4c-kube-api-access-8pm6m\") pod \"dbb213fc-764a-4f5b-9866-043e798f6a4c\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " Nov 03 22:56:23.200151 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.200040 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-openshift-ca\") pod \"dbb213fc-764a-4f5b-9866-043e798f6a4c\" (UID: \"dbb213fc-764a-4f5b-9866-043e798f6a4c\") " Nov 03 22:56:23.200466 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.200436 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-openshift-ca" pod "dbb213fc-764a-4f5b-9866-043e798f6a4c" (UID: "dbb213fc-764a-4f5b-9866-043e798f6a4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:23.202068 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.202039 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-xks-ca" pod "dbb213fc-764a-4f5b-9866-043e798f6a4c" (UID: "dbb213fc-764a-4f5b-9866-043e798f6a4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:23.202171 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.202152 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb213fc-764a-4f5b-9866-043e798f6a4c-kube-api-access-8pm6m" pod "dbb213fc-764a-4f5b-9866-043e798f6a4c" (UID: "dbb213fc-764a-4f5b-9866-043e798f6a4c"). InnerVolumeSpecName "kube-api-access-8pm6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:23.301603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.301572 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:23.301603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.301597 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pm6m\" (UniqueName: \"kubernetes.io/projected/dbb213fc-764a-4f5b-9866-043e798f6a4c-kube-api-access-8pm6m\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:23.301603 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.301610 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb213fc-764a-4f5b-9866-043e798f6a4c-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:23.826490 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.826462 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" Nov 03 22:56:23.826892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.826531 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z" event={"ID":"dbb213fc-764a-4f5b-9866-043e798f6a4c","Type":"ContainerDied","Data":"1793f78dbbbc510a028a1ca713a78539593c4039515973003f3eac352454c1ab"} Nov 03 22:56:23.826892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:23.826574 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1793f78dbbbc510a028a1ca713a78539593c4039515973003f3eac352454c1ab" Nov 03 22:56:24.602897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:24.602864 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:27.500443 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:27.500397 4298 scope.go:122] "RemoveContainer" containerID="1a7c1d73089b67a633235ebc980a32a7369f5f757b084894a5637b7661b82541" Nov 03 22:56:27.509183 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:27.509162 4298 scope.go:122] "RemoveContainer" containerID="294f41e457ec4beb90506da4508d94fd06b7b8582b3ac0f4fb38d59c3bea1fd3" Nov 03 22:56:30.159611 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:30.159580 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:47.551638 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:56:47.551598 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:56:58.276016 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.275982 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5"] Nov 03 22:56:58.276521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.276394 4298 cpu_manager.go:414] "containerMap: removing container" podUID="dbb213fc-764a-4f5b-9866-043e798f6a4c" containerName="revoke-keys" Nov 03 22:56:58.276521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.276405 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb213fc-764a-4f5b-9866-043e798f6a4c" containerName="revoke-keys" Nov 03 22:56:58.276521 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.276490 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb213fc-764a-4f5b-9866-043e798f6a4c" containerName="revoke-keys" Nov 03 22:56:58.279710 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.279693 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:56:58.354216 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.354182 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vhhd5\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:56:58.354412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.354243 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vhhd5\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:56:58.354412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.354352 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcnqh\" (UniqueName: \"kubernetes.io/projected/849d7411-37d7-44c7-a419-0e0a9f10b59f-kube-api-access-mcnqh\") pod \"maas-api-revoke-sub-046e1343-vhhd5\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:56:58.383456 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.383420 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:56:58.383674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.383485 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:56:58.386574 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.386554 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5"] Nov 03 22:56:58.454856 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.454830 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vhhd5\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:56:58.455031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.454879 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vhhd5\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:56:58.455031 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.454911 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcnqh\" (UniqueName: \"kubernetes.io/projected/849d7411-37d7-44c7-a419-0e0a9f10b59f-kube-api-access-mcnqh\") pod \"maas-api-revoke-sub-046e1343-vhhd5\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:56:58.455489 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.455467 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vhhd5\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:56:58.457244 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.457222 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vhhd5\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:56:58.511840 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.511809 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcnqh\" (UniqueName: \"kubernetes.io/projected/849d7411-37d7-44c7-a419-0e0a9f10b59f-kube-api-access-mcnqh\") pod \"maas-api-revoke-sub-046e1343-vhhd5\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:56:58.592411 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.592316 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:56:58.955081 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:56:58.955048 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849d7411_37d7_44c7_a419_0e0a9f10b59f.slice/crio-62c73f34298e1e06502e81304db34a7ab4b128f9d6f0c5efb0017386f3cc75f1 WatchSource:0}: Error finding container 62c73f34298e1e06502e81304db34a7ab4b128f9d6f0c5efb0017386f3cc75f1: Status 404 returned error can't find the container with id 62c73f34298e1e06502e81304db34a7ab4b128f9d6f0c5efb0017386f3cc75f1 Nov 03 22:56:58.959881 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:58.959858 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5"] Nov 03 22:56:59.027426 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:56:59.027390 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" event={"ID":"849d7411-37d7-44c7-a419-0e0a9f10b59f","Type":"ContainerStarted","Data":"62c73f34298e1e06502e81304db34a7ab4b128f9d6f0c5efb0017386f3cc75f1"} Nov 03 22:57:00.034564 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:00.034529 4298 generic.go:356] "Generic (PLEG): container finished" podID="849d7411-37d7-44c7-a419-0e0a9f10b59f" containerID="610150db945598b422f235376f44c294b0489c55efa86b66f5aa8632b98a667c" exitCode=0 Nov 03 22:57:00.035029 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:00.034612 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" event={"ID":"849d7411-37d7-44c7-a419-0e0a9f10b59f","Type":"ContainerDied","Data":"610150db945598b422f235376f44c294b0489c55efa86b66f5aa8632b98a667c"} Nov 03 22:57:01.177928 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:01.177897 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:57:01.284367 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:01.284333 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-openshift-ca\") pod \"849d7411-37d7-44c7-a419-0e0a9f10b59f\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " Nov 03 22:57:01.284576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:01.284377 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-xks-ca\") pod \"849d7411-37d7-44c7-a419-0e0a9f10b59f\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " Nov 03 22:57:01.284576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:01.284404 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849d7411-37d7-44c7-a419-0e0a9f10b59f-kube-api-access-mcnqh\" (UniqueName: \"kubernetes.io/projected/849d7411-37d7-44c7-a419-0e0a9f10b59f-kube-api-access-mcnqh\") pod \"849d7411-37d7-44c7-a419-0e0a9f10b59f\" (UID: \"849d7411-37d7-44c7-a419-0e0a9f10b59f\") " Nov 03 22:57:01.284836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:01.284810 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-openshift-ca" pod "849d7411-37d7-44c7-a419-0e0a9f10b59f" (UID: "849d7411-37d7-44c7-a419-0e0a9f10b59f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:01.286529 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:01.286501 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849d7411-37d7-44c7-a419-0e0a9f10b59f-kube-api-access-mcnqh" pod "849d7411-37d7-44c7-a419-0e0a9f10b59f" (UID: "849d7411-37d7-44c7-a419-0e0a9f10b59f"). InnerVolumeSpecName "kube-api-access-mcnqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:01.286656 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:01.286577 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-xks-ca" pod "849d7411-37d7-44c7-a419-0e0a9f10b59f" (UID: "849d7411-37d7-44c7-a419-0e0a9f10b59f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:01.385357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:01.385316 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:01.385357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:01.385352 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849d7411-37d7-44c7-a419-0e0a9f10b59f-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:01.385357 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:01.385364 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcnqh\" (UniqueName: \"kubernetes.io/projected/849d7411-37d7-44c7-a419-0e0a9f10b59f-kube-api-access-mcnqh\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:02.047260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:02.047223 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" Nov 03 22:57:02.047260 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:02.047254 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5" event={"ID":"849d7411-37d7-44c7-a419-0e0a9f10b59f","Type":"ContainerDied","Data":"62c73f34298e1e06502e81304db34a7ab4b128f9d6f0c5efb0017386f3cc75f1"} Nov 03 22:57:02.047530 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:02.047335 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62c73f34298e1e06502e81304db34a7ab4b128f9d6f0c5efb0017386f3cc75f1" Nov 03 22:57:09.055790 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.055757 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn"] Nov 03 22:57:09.056384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.056364 4298 cpu_manager.go:414] "containerMap: removing container" podUID="849d7411-37d7-44c7-a419-0e0a9f10b59f" containerName="revoke-keys" Nov 03 22:57:09.056483 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.056386 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849d7411-37d7-44c7-a419-0e0a9f10b59f" containerName="revoke-keys" Nov 03 22:57:09.056543 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.056480 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="849d7411-37d7-44c7-a419-0e0a9f10b59f" containerName="revoke-keys" Nov 03 22:57:09.059892 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.059868 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:09.160675 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.160626 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-m5hdn\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:09.160870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.160745 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7bn\" (UniqueName: \"kubernetes.io/projected/f7be1a2a-6348-4909-b059-d0c40df63e99-kube-api-access-wk7bn\") pod \"maas-api-revoke-sub-d3fba056-m5hdn\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:09.160870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.160777 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-m5hdn\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:09.163117 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.163091 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:57:09.163241 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.163141 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:57:09.166333 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.166311 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn"] Nov 03 22:57:09.261860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.261823 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7bn\" (UniqueName: \"kubernetes.io/projected/f7be1a2a-6348-4909-b059-d0c40df63e99-kube-api-access-wk7bn\") pod \"maas-api-revoke-sub-d3fba056-m5hdn\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:09.261860 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.261860 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-m5hdn\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:09.262049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.261929 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-m5hdn\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:09.262531 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.262504 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-m5hdn\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:09.264315 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.264292 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-m5hdn\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:09.318787 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.318712 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7bn\" (UniqueName: \"kubernetes.io/projected/f7be1a2a-6348-4909-b059-d0c40df63e99-kube-api-access-wk7bn\") pod \"maas-api-revoke-sub-d3fba056-m5hdn\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:09.373164 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.373126 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:09.737122 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:57:09.737090 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7be1a2a_6348_4909_b059_d0c40df63e99.slice/crio-3ea0c83cddbfb478475a4140f251b0b697155e441aa5fe93d6737d8c4026454a WatchSource:0}: Error finding container 3ea0c83cddbfb478475a4140f251b0b697155e441aa5fe93d6737d8c4026454a: Status 404 returned error can't find the container with id 3ea0c83cddbfb478475a4140f251b0b697155e441aa5fe93d6737d8c4026454a Nov 03 22:57:09.740555 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:09.740533 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn"] Nov 03 22:57:10.092337 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:10.092297 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" event={"ID":"f7be1a2a-6348-4909-b059-d0c40df63e99","Type":"ContainerStarted","Data":"92026a68a4aede96e6aa7cd79c62874051cb6df52f1de97d4ba9ef7bd20a1dbc"} Nov 03 22:57:10.092761 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:10.092371 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" event={"ID":"f7be1a2a-6348-4909-b059-d0c40df63e99","Type":"ContainerStarted","Data":"3ea0c83cddbfb478475a4140f251b0b697155e441aa5fe93d6737d8c4026454a"} Nov 03 22:57:10.207897 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:10.207837 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" podStartSLOduration=1.207824072 podStartE2EDuration="1.207824072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:10.205221964 +0000 UTC m=+2987.185955616" watchObservedRunningTime="2026-11-03 22:57:10.207824072 +0000 UTC m=+2987.188557725" Nov 03 22:57:11.099359 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:11.099324 4298 generic.go:356] "Generic (PLEG): container finished" podID="f7be1a2a-6348-4909-b059-d0c40df63e99" containerID="92026a68a4aede96e6aa7cd79c62874051cb6df52f1de97d4ba9ef7bd20a1dbc" exitCode=0 Nov 03 22:57:11.099779 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:11.099407 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" event={"ID":"f7be1a2a-6348-4909-b059-d0c40df63e99","Type":"ContainerDied","Data":"92026a68a4aede96e6aa7cd79c62874051cb6df52f1de97d4ba9ef7bd20a1dbc"} Nov 03 22:57:12.258460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:12.258425 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:12.404882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:12.403950 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-openshift-ca\") pod \"f7be1a2a-6348-4909-b059-d0c40df63e99\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " Nov 03 22:57:12.404882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:12.404083 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7be1a2a-6348-4909-b059-d0c40df63e99-kube-api-access-wk7bn\" (UniqueName: \"kubernetes.io/projected/f7be1a2a-6348-4909-b059-d0c40df63e99-kube-api-access-wk7bn\") pod \"f7be1a2a-6348-4909-b059-d0c40df63e99\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " Nov 03 22:57:12.404882 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:12.404159 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-xks-ca\") pod \"f7be1a2a-6348-4909-b059-d0c40df63e99\" (UID: \"f7be1a2a-6348-4909-b059-d0c40df63e99\") " Nov 03 22:57:12.405889 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:12.405864 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-openshift-ca" pod "f7be1a2a-6348-4909-b059-d0c40df63e99" (UID: "f7be1a2a-6348-4909-b059-d0c40df63e99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:12.409253 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:12.409192 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-xks-ca" pod "f7be1a2a-6348-4909-b059-d0c40df63e99" (UID: "f7be1a2a-6348-4909-b059-d0c40df63e99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:12.410165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:12.410144 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7be1a2a-6348-4909-b059-d0c40df63e99-kube-api-access-wk7bn" pod "f7be1a2a-6348-4909-b059-d0c40df63e99" (UID: "f7be1a2a-6348-4909-b059-d0c40df63e99"). InnerVolumeSpecName "kube-api-access-wk7bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:12.505559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:12.505518 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:12.505559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:12.505553 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be1a2a-6348-4909-b059-d0c40df63e99-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:12.505559 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:12.505566 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7bn\" (UniqueName: \"kubernetes.io/projected/f7be1a2a-6348-4909-b059-d0c40df63e99-kube-api-access-wk7bn\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:13.112784 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:13.112744 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" event={"ID":"f7be1a2a-6348-4909-b059-d0c40df63e99","Type":"ContainerDied","Data":"3ea0c83cddbfb478475a4140f251b0b697155e441aa5fe93d6737d8c4026454a"} Nov 03 22:57:13.112975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:13.112819 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ea0c83cddbfb478475a4140f251b0b697155e441aa5fe93d6737d8c4026454a" Nov 03 22:57:13.112975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:13.112753 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn" Nov 03 22:57:16.047308 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:16.047271 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst"] Nov 03 22:57:16.097770 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:16.097729 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc0b00f1-vnfst"] Nov 03 22:57:17.047660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:17.047610 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb"] Nov 03 22:57:17.098063 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:17.098030 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d9df21-fdqqb"] Nov 03 22:57:17.134515 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:17.134486 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a19f04-01d5-4af7-aff6-22f1d4bc3d20" path="/var/lib/kubelet/pods/33a19f04-01d5-4af7-aff6-22f1d4bc3d20/volumes" Nov 03 22:57:17.135018 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:17.134998 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58871bb6-bc1c-42ad-9432-9bf28802fa2a" path="/var/lib/kubelet/pods/58871bb6-bc1c-42ad-9432-9bf28802fa2a/volumes" Nov 03 22:57:17.551462 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:57:17.551429 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:57:20.340836 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:20.340803 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:24.907662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:24.907616 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:27.533249 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:27.533218 4298 scope.go:122] "RemoveContainer" containerID="fb6a5dc1cde91eb12561ef7cc1dc41388946e6b7db6d071c3db07adf649564f6" Nov 03 22:57:27.542050 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:27.542031 4298 scope.go:122] "RemoveContainer" containerID="29b06420657ca25883d3ba52463d0565c81f782eada869842d7d65dac8d3bc06" Nov 03 22:57:27.549751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:27.549730 4298 scope.go:122] "RemoveContainer" containerID="e66617d2c97d937e5da893d49bc100435d0b359b53c2167e60422d561337631c" Nov 03 22:57:27.557549 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:27.557530 4298 scope.go:122] "RemoveContainer" containerID="a666a4b0fa3c7f7bbb3fa5855500a109b359796b0137a097f5526a7084115c0b" Nov 03 22:57:27.564859 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:27.564841 4298 scope.go:122] "RemoveContainer" containerID="13274c397259e47b01124e560c8bb0e73334df7d7d879a9941362692311425dd" Nov 03 22:57:27.572141 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:27.572124 4298 scope.go:122] "RemoveContainer" containerID="9bc4c4f8ca61f491e5d862bb74ff076f14fd0d4c6b6fdd141c17aa22267bd520" Nov 03 22:57:27.579181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:27.579166 4298 scope.go:122] "RemoveContainer" containerID="0dc30e18b2739a8761c32079b9fd5ba2c151fe39908b2374ce851074b49dea0f" Nov 03 22:57:30.592861 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:30.592787 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:43.969240 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:43.969162 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz"] Nov 03 22:57:43.969622 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:43.969536 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f7be1a2a-6348-4909-b059-d0c40df63e99" containerName="revoke-keys" Nov 03 22:57:43.969622 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:43.969547 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7be1a2a-6348-4909-b059-d0c40df63e99" containerName="revoke-keys" Nov 03 22:57:43.969725 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:43.969635 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7be1a2a-6348-4909-b059-d0c40df63e99" containerName="revoke-keys" Nov 03 22:57:43.974425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:43.974408 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:44.030139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.030102 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6vc\" (UniqueName: \"kubernetes.io/projected/ad94ffff-f952-41db-b992-1ee7c97af223-kube-api-access-jv6vc\") pod \"maas-api-revoke-sub-ae594979-4nzjz\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:44.030139 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.030148 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4nzjz\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:44.030394 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.030366 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4nzjz\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:44.030520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.030498 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz"] Nov 03 22:57:44.076460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.076431 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:57:44.076628 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.076492 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:57:44.131181 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.131138 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4nzjz\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:44.131380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.131205 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6vc\" (UniqueName: \"kubernetes.io/projected/ad94ffff-f952-41db-b992-1ee7c97af223-kube-api-access-jv6vc\") pod \"maas-api-revoke-sub-ae594979-4nzjz\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:44.131380 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.131226 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4nzjz\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:44.131752 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.131729 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4nzjz\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:44.133523 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.133500 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4nzjz\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:44.188677 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.188632 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6vc\" (UniqueName: \"kubernetes.io/projected/ad94ffff-f952-41db-b992-1ee7c97af223-kube-api-access-jv6vc\") pod \"maas-api-revoke-sub-ae594979-4nzjz\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:44.286863 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.286775 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:44.651558 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:57:44.651530 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad94ffff_f952_41db_b992_1ee7c97af223.slice/crio-36b94de1593f3e0bb9f7052de7d727367fd81c327eda9571baafb311327bb680 WatchSource:0}: Error finding container 36b94de1593f3e0bb9f7052de7d727367fd81c327eda9571baafb311327bb680: Status 404 returned error can't find the container with id 36b94de1593f3e0bb9f7052de7d727367fd81c327eda9571baafb311327bb680 Nov 03 22:57:44.654557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:44.654537 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz"] Nov 03 22:57:45.283886 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:45.283845 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" event={"ID":"ad94ffff-f952-41db-b992-1ee7c97af223","Type":"ContainerStarted","Data":"54cfe5b2c090332cd218d491922a7a6a34f4cc3434f6f7f27058ea0044536fef"} Nov 03 22:57:45.284322 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:45.283939 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" event={"ID":"ad94ffff-f952-41db-b992-1ee7c97af223","Type":"ContainerStarted","Data":"36b94de1593f3e0bb9f7052de7d727367fd81c327eda9571baafb311327bb680"} Nov 03 22:57:45.414662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:45.414594 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" podStartSLOduration=2.414580967 podStartE2EDuration="2.414580967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:45.396337128 +0000 UTC m=+3022.377070780" watchObservedRunningTime="2026-11-03 22:57:45.414580967 +0000 UTC m=+3022.395314653" Nov 03 22:57:46.290566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:46.290527 4298 generic.go:356] "Generic (PLEG): container finished" podID="ad94ffff-f952-41db-b992-1ee7c97af223" containerID="54cfe5b2c090332cd218d491922a7a6a34f4cc3434f6f7f27058ea0044536fef" exitCode=0 Nov 03 22:57:46.291028 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:46.290613 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" event={"ID":"ad94ffff-f952-41db-b992-1ee7c97af223","Type":"ContainerDied","Data":"54cfe5b2c090332cd218d491922a7a6a34f4cc3434f6f7f27058ea0044536fef"} Nov 03 22:57:47.439129 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:47.439104 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:47.551186 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:57:47.551099 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:57:47.565345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:47.565315 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-xks-ca\") pod \"ad94ffff-f952-41db-b992-1ee7c97af223\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " Nov 03 22:57:47.565542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:47.565385 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-openshift-ca\") pod \"ad94ffff-f952-41db-b992-1ee7c97af223\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " Nov 03 22:57:47.565542 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:47.565420 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad94ffff-f952-41db-b992-1ee7c97af223-kube-api-access-jv6vc\" (UniqueName: \"kubernetes.io/projected/ad94ffff-f952-41db-b992-1ee7c97af223-kube-api-access-jv6vc\") pod \"ad94ffff-f952-41db-b992-1ee7c97af223\" (UID: \"ad94ffff-f952-41db-b992-1ee7c97af223\") " Nov 03 22:57:47.565743 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:47.565713 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-openshift-ca" pod "ad94ffff-f952-41db-b992-1ee7c97af223" (UID: "ad94ffff-f952-41db-b992-1ee7c97af223"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:47.567487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:47.567463 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-xks-ca" pod "ad94ffff-f952-41db-b992-1ee7c97af223" (UID: "ad94ffff-f952-41db-b992-1ee7c97af223"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:47.567487 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:47.567473 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad94ffff-f952-41db-b992-1ee7c97af223-kube-api-access-jv6vc" pod "ad94ffff-f952-41db-b992-1ee7c97af223" (UID: "ad94ffff-f952-41db-b992-1ee7c97af223"). InnerVolumeSpecName "kube-api-access-jv6vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:47.666980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:47.666938 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:47.666980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:47.666969 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad94ffff-f952-41db-b992-1ee7c97af223-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:47.666980 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:47.666981 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv6vc\" (UniqueName: \"kubernetes.io/projected/ad94ffff-f952-41db-b992-1ee7c97af223-kube-api-access-jv6vc\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:48.302921 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:48.302897 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" Nov 03 22:57:48.303126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:48.302890 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz" event={"ID":"ad94ffff-f952-41db-b992-1ee7c97af223","Type":"ContainerDied","Data":"36b94de1593f3e0bb9f7052de7d727367fd81c327eda9571baafb311327bb680"} Nov 03 22:57:48.303126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:48.303025 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36b94de1593f3e0bb9f7052de7d727367fd81c327eda9571baafb311327bb680" Nov 03 22:57:50.048460 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:50.048418 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5"] Nov 03 22:57:50.098877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:50.098841 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de649233-8jzt5"] Nov 03 22:57:50.101576 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:50.101550 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p"] Nov 03 22:57:50.104082 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:50.104061 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-145293ea-wcp5p"] Nov 03 22:57:51.134412 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:51.134364 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1accc7f5-0255-43de-8ce1-5f892f8ff521" path="/var/lib/kubelet/pods/1accc7f5-0255-43de-8ce1-5f892f8ff521/volumes" Nov 03 22:57:51.135686 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:57:51.135638 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6" path="/var/lib/kubelet/pods/b2cc5baa-8a68-442b-9ceb-1ce6016e2ca6/volumes" Nov 03 22:58:12.234660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.234611 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg"] Nov 03 22:58:12.237178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.235023 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ad94ffff-f952-41db-b992-1ee7c97af223" containerName="revoke-keys" Nov 03 22:58:12.237178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.235035 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad94ffff-f952-41db-b992-1ee7c97af223" containerName="revoke-keys" Nov 03 22:58:12.237178 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.235101 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad94ffff-f952-41db-b992-1ee7c97af223" containerName="revoke-keys" Nov 03 22:58:12.238210 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.238195 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:12.292454 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.292425 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:58:12.296977 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.296933 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg"] Nov 03 22:58:12.315557 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.315527 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bkscg\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:12.315745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.315577 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw2g\" (UniqueName: \"kubernetes.io/projected/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-kube-api-access-cfw2g\") pod \"maas-api-revoke-sub-a4aef7d7-bkscg\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:12.315745 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.315716 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bkscg\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:12.342796 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.342774 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:58:12.416551 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.416509 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bkscg\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:12.416794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.416575 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw2g\" (UniqueName: \"kubernetes.io/projected/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-kube-api-access-cfw2g\") pod \"maas-api-revoke-sub-a4aef7d7-bkscg\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:12.416794 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.416623 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bkscg\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:12.417223 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.417202 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bkscg\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:12.418964 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.418941 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bkscg\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:12.474168 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.474135 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw2g\" (UniqueName: \"kubernetes.io/projected/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-kube-api-access-cfw2g\") pod \"maas-api-revoke-sub-a4aef7d7-bkscg\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:12.551167 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.551076 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:12.914460 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:58:12.914421 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3158106_e93e_4dfe_84cd_a7a8d7d074f3.slice/crio-a5e710326814408ca7f7d5faa447368099de8d4488dd213d2f2972f6dd087b93 WatchSource:0}: Error finding container a5e710326814408ca7f7d5faa447368099de8d4488dd213d2f2972f6dd087b93: Status 404 returned error can't find the container with id a5e710326814408ca7f7d5faa447368099de8d4488dd213d2f2972f6dd087b93 Nov 03 22:58:12.917261 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:12.917235 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg"] Nov 03 22:58:13.441705 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:13.441671 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" event={"ID":"e3158106-e93e-4dfe-84cd-a7a8d7d074f3","Type":"ContainerStarted","Data":"4f6f1645d5c284dc0bb6d0005df4658816c8492d57341b203c3f200af06ef3a9"} Nov 03 22:58:13.442097 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:13.441728 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" event={"ID":"e3158106-e93e-4dfe-84cd-a7a8d7d074f3","Type":"ContainerStarted","Data":"a5e710326814408ca7f7d5faa447368099de8d4488dd213d2f2972f6dd087b93"} Nov 03 22:58:13.558173 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:13.558057 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" podStartSLOduration=1.558042673 podStartE2EDuration="1.558042673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:13.555758109 +0000 UTC m=+3050.536491761" watchObservedRunningTime="2026-11-03 22:58:13.558042673 +0000 UTC m=+3050.538776325" Nov 03 22:58:14.447621 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:14.447587 4298 generic.go:356] "Generic (PLEG): container finished" podID="e3158106-e93e-4dfe-84cd-a7a8d7d074f3" containerID="4f6f1645d5c284dc0bb6d0005df4658816c8492d57341b203c3f200af06ef3a9" exitCode=0 Nov 03 22:58:14.448037 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:14.447652 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" event={"ID":"e3158106-e93e-4dfe-84cd-a7a8d7d074f3","Type":"ContainerDied","Data":"4f6f1645d5c284dc0bb6d0005df4658816c8492d57341b203c3f200af06ef3a9"} Nov 03 22:58:15.576614 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:15.576589 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:15.646410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:15.646385 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-xks-ca\") pod \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " Nov 03 22:58:15.646566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:15.646450 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-openshift-ca\") pod \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " Nov 03 22:58:15.646566 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:15.646540 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-kube-api-access-cfw2g\" (UniqueName: \"kubernetes.io/projected/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-kube-api-access-cfw2g\") pod \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\" (UID: \"e3158106-e93e-4dfe-84cd-a7a8d7d074f3\") " Nov 03 22:58:15.646899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:15.646873 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-openshift-ca" pod "e3158106-e93e-4dfe-84cd-a7a8d7d074f3" (UID: "e3158106-e93e-4dfe-84cd-a7a8d7d074f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:15.648539 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:15.648513 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-xks-ca" pod "e3158106-e93e-4dfe-84cd-a7a8d7d074f3" (UID: "e3158106-e93e-4dfe-84cd-a7a8d7d074f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:15.648631 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:15.648533 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-kube-api-access-cfw2g" pod "e3158106-e93e-4dfe-84cd-a7a8d7d074f3" (UID: "e3158106-e93e-4dfe-84cd-a7a8d7d074f3"). InnerVolumeSpecName "kube-api-access-cfw2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:15.748036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:15.747957 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfw2g\" (UniqueName: \"kubernetes.io/projected/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-kube-api-access-cfw2g\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:15.748036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:15.747984 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:15.748036 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:15.747994 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3158106-e93e-4dfe-84cd-a7a8d7d074f3-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:16.458165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:16.458135 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" Nov 03 22:58:16.458430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:16.458171 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg" event={"ID":"e3158106-e93e-4dfe-84cd-a7a8d7d074f3","Type":"ContainerDied","Data":"a5e710326814408ca7f7d5faa447368099de8d4488dd213d2f2972f6dd087b93"} Nov 03 22:58:16.458430 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:16.458220 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e710326814408ca7f7d5faa447368099de8d4488dd213d2f2972f6dd087b93" Nov 03 22:58:17.551668 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:58:17.551615 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:58:20.731352 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:20.731322 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:25.316662 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:25.316621 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:27.622218 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:27.622129 4298 scope.go:122] "RemoveContainer" containerID="737eb7fa10f07c7b215bf9708c959c9985d96fd304f8294583f0e340a7313696" Nov 03 22:58:27.632877 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:27.632853 4298 scope.go:122] "RemoveContainer" containerID="1d6dad44df1aa83aae24a97e4ff0542e871a9b89f1214c2552bba16fac8bf450" Nov 03 22:58:27.640915 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:27.640895 4298 scope.go:122] "RemoveContainer" containerID="397d4faae2d895a2a8d787aa304107f869ea0a272bd9b03763f12df21c525c47" Nov 03 22:58:27.648690 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:27.648671 4298 scope.go:122] "RemoveContainer" containerID="1ab7df3e72607640a48f960c94780f8ee7b629dbb64bd3c0c559464d96654b05" Nov 03 22:58:27.656109 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:27.656083 4298 scope.go:122] "RemoveContainer" containerID="42b8fd64d2505b368feffa7e6b20497cc25671bbfd1e22405cdd75fe678fde56" Nov 03 22:58:27.663581 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:27.663565 4298 scope.go:122] "RemoveContainer" containerID="d57b4e5c6e14ad9fc8cd549ba0bccd3922c23f503ffc0799299ecb653be44469" Nov 03 22:58:27.670868 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:27.670847 4298 scope.go:122] "RemoveContainer" containerID="8035f14d0bab5038e96a4b4b5590b0dcf788b3ecf354fcbfa8a606f403727eea" Nov 03 22:58:30.963049 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:30.963017 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:39.050263 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:39.049826 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v"] Nov 03 22:58:39.098087 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:39.098053 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d19310-mgj8v"] Nov 03 22:58:39.134207 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:39.134176 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46a7ddb-9ca7-4b6f-bc1b-43d00109d759" path="/var/lib/kubelet/pods/d46a7ddb-9ca7-4b6f-bc1b-43d00109d759/volumes" Nov 03 22:58:42.966923 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:42.966889 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4"] Nov 03 22:58:42.967345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:42.967319 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e3158106-e93e-4dfe-84cd-a7a8d7d074f3" containerName="revoke-keys" Nov 03 22:58:42.967345 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:42.967336 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3158106-e93e-4dfe-84cd-a7a8d7d074f3" containerName="revoke-keys" Nov 03 22:58:42.967425 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:42.967402 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3158106-e93e-4dfe-84cd-a7a8d7d074f3" containerName="revoke-keys" Nov 03 22:58:42.970813 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:42.970797 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:43.024327 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.024305 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:58:43.074447 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.074419 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:58:43.077594 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.077560 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4"] Nov 03 22:58:43.130180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.130150 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pnlp\" (UniqueName: \"kubernetes.io/projected/f46837d1-667e-49c3-906f-4bdad7520a48-kube-api-access-5pnlp\") pod \"maas-api-revoke-sub-79805ca9-m6hc4\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:43.130398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.130284 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-m6hc4\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:43.130398 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.130328 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-m6hc4\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:43.231696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.231584 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pnlp\" (UniqueName: \"kubernetes.io/projected/f46837d1-667e-49c3-906f-4bdad7520a48-kube-api-access-5pnlp\") pod \"maas-api-revoke-sub-79805ca9-m6hc4\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:43.231696 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.231695 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-m6hc4\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:43.231959 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.231734 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-m6hc4\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:43.232375 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.232343 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-m6hc4\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:43.234060 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.234042 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-m6hc4\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:43.290956 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.290929 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pnlp\" (UniqueName: \"kubernetes.io/projected/f46837d1-667e-49c3-906f-4bdad7520a48-kube-api-access-5pnlp\") pod \"maas-api-revoke-sub-79805ca9-m6hc4\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:43.583328 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.583245 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:43.952912 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:58:43.952877 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46837d1_667e_49c3_906f_4bdad7520a48.slice/crio-29b2acec608cafb71bd7d58652ed1db3d80c69130b92ab221063a9ee9a4d389c WatchSource:0}: Error finding container 29b2acec608cafb71bd7d58652ed1db3d80c69130b92ab221063a9ee9a4d389c: Status 404 returned error can't find the container with id 29b2acec608cafb71bd7d58652ed1db3d80c69130b92ab221063a9ee9a4d389c Nov 03 22:58:43.955061 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:43.955038 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4"] Nov 03 22:58:44.612535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:44.612441 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" event={"ID":"f46837d1-667e-49c3-906f-4bdad7520a48","Type":"ContainerStarted","Data":"8c14cabdd485ac802ad2c018d091ba7ea99e8d55594211b6ac04f5a25aead3da"} Nov 03 22:58:44.612535 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:44.612500 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" event={"ID":"f46837d1-667e-49c3-906f-4bdad7520a48","Type":"ContainerStarted","Data":"29b2acec608cafb71bd7d58652ed1db3d80c69130b92ab221063a9ee9a4d389c"} Nov 03 22:58:44.728660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:44.728603 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" podStartSLOduration=2.728589812 podStartE2EDuration="2.728589812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:44.727147408 +0000 UTC m=+3081.707881060" watchObservedRunningTime="2026-11-03 22:58:44.728589812 +0000 UTC m=+3081.709323466" Nov 03 22:58:45.619180 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:45.619142 4298 generic.go:356] "Generic (PLEG): container finished" podID="f46837d1-667e-49c3-906f-4bdad7520a48" containerID="8c14cabdd485ac802ad2c018d091ba7ea99e8d55594211b6ac04f5a25aead3da" exitCode=0 Nov 03 22:58:45.619692 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:45.619241 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" event={"ID":"f46837d1-667e-49c3-906f-4bdad7520a48","Type":"ContainerDied","Data":"8c14cabdd485ac802ad2c018d091ba7ea99e8d55594211b6ac04f5a25aead3da"} Nov 03 22:58:46.756421 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:46.756396 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:46.869971 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:46.869935 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-xks-ca\") pod \"f46837d1-667e-49c3-906f-4bdad7520a48\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " Nov 03 22:58:46.870184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:46.869998 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-openshift-ca\") pod \"f46837d1-667e-49c3-906f-4bdad7520a48\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " Nov 03 22:58:46.870184 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:46.870034 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46837d1-667e-49c3-906f-4bdad7520a48-kube-api-access-5pnlp\" (UniqueName: \"kubernetes.io/projected/f46837d1-667e-49c3-906f-4bdad7520a48-kube-api-access-5pnlp\") pod \"f46837d1-667e-49c3-906f-4bdad7520a48\" (UID: \"f46837d1-667e-49c3-906f-4bdad7520a48\") " Nov 03 22:58:46.870372 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:46.870347 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-openshift-ca" pod "f46837d1-667e-49c3-906f-4bdad7520a48" (UID: "f46837d1-667e-49c3-906f-4bdad7520a48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:46.872403 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:46.872372 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46837d1-667e-49c3-906f-4bdad7520a48-kube-api-access-5pnlp" pod "f46837d1-667e-49c3-906f-4bdad7520a48" (UID: "f46837d1-667e-49c3-906f-4bdad7520a48"). InnerVolumeSpecName "kube-api-access-5pnlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:46.872568 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:46.872461 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-xks-ca" pod "f46837d1-667e-49c3-906f-4bdad7520a48" (UID: "f46837d1-667e-49c3-906f-4bdad7520a48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:46.971302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:46.971264 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.971302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:46.971298 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46837d1-667e-49c3-906f-4bdad7520a48-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.971302 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:46.971311 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pnlp\" (UniqueName: \"kubernetes.io/projected/f46837d1-667e-49c3-906f-4bdad7520a48-kube-api-access-5pnlp\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:47.552124 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:58:47.552085 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:58:47.630694 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:47.630662 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" Nov 03 22:58:47.630887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:47.630694 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4" event={"ID":"f46837d1-667e-49c3-906f-4bdad7520a48","Type":"ContainerDied","Data":"29b2acec608cafb71bd7d58652ed1db3d80c69130b92ab221063a9ee9a4d389c"} Nov 03 22:58:47.630887 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:58:47.630748 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29b2acec608cafb71bd7d58652ed1db3d80c69130b92ab221063a9ee9a4d389c" Nov 03 22:59:03.678970 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.678884 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n"] Nov 03 22:59:03.679376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.679304 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f46837d1-667e-49c3-906f-4bdad7520a48" containerName="revoke-keys" Nov 03 22:59:03.679376 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.679316 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46837d1-667e-49c3-906f-4bdad7520a48" containerName="revoke-keys" Nov 03 22:59:03.679470 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.679397 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46837d1-667e-49c3-906f-4bdad7520a48" containerName="revoke-keys" Nov 03 22:59:03.682401 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.682384 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:03.740165 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.740127 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n"] Nov 03 22:59:03.744170 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.744133 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qwf8n\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:03.744368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.744185 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8m6\" (UniqueName: \"kubernetes.io/projected/e9d87073-2802-41e4-9d83-bc318c44cb87-kube-api-access-nd8m6\") pod \"maas-api-revoke-sub-0b7e8edb-qwf8n\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:03.744368 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.744212 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qwf8n\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:03.786313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.786276 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:59:03.786313 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.786278 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:59:03.845448 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.845413 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qwf8n\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:03.845727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.845459 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8m6\" (UniqueName: \"kubernetes.io/projected/e9d87073-2802-41e4-9d83-bc318c44cb87-kube-api-access-nd8m6\") pod \"maas-api-revoke-sub-0b7e8edb-qwf8n\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:03.845727 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.845484 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qwf8n\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:03.846134 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.846099 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qwf8n\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:03.848094 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.848068 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qwf8n\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:03.902801 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.902760 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8m6\" (UniqueName: \"kubernetes.io/projected/e9d87073-2802-41e4-9d83-bc318c44cb87-kube-api-access-nd8m6\") pod \"maas-api-revoke-sub-0b7e8edb-qwf8n\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:03.995179 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:03.995144 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:04.362171 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:59:04.362136 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d87073_2802_41e4_9d83_bc318c44cb87.slice/crio-99fdf9bcd852c04e469c4ec67b00a5f6e485f9f4e636217707a95c98bbeb16ae WatchSource:0}: Error finding container 99fdf9bcd852c04e469c4ec67b00a5f6e485f9f4e636217707a95c98bbeb16ae: Status 404 returned error can't find the container with id 99fdf9bcd852c04e469c4ec67b00a5f6e485f9f4e636217707a95c98bbeb16ae Nov 03 22:59:04.364808 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:04.364669 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n"] Nov 03 22:59:04.727381 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:04.727344 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" event={"ID":"e9d87073-2802-41e4-9d83-bc318c44cb87","Type":"ContainerStarted","Data":"8dd13e078142ad49d1217f28e8e716bf4d2cea4013065a709cc52b52279cbcdc"} Nov 03 22:59:04.727823 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:04.727399 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" event={"ID":"e9d87073-2802-41e4-9d83-bc318c44cb87","Type":"ContainerStarted","Data":"99fdf9bcd852c04e469c4ec67b00a5f6e485f9f4e636217707a95c98bbeb16ae"} Nov 03 22:59:04.850693 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:04.844835 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" podStartSLOduration=1.8448176059999999 podStartE2EDuration="1.844817606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:04.840687667 +0000 UTC m=+3101.821421320" watchObservedRunningTime="2026-11-03 22:59:04.844817606 +0000 UTC m=+3101.825551261" Nov 03 22:59:05.733434 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:05.733397 4298 generic.go:356] "Generic (PLEG): container finished" podID="e9d87073-2802-41e4-9d83-bc318c44cb87" containerID="8dd13e078142ad49d1217f28e8e716bf4d2cea4013065a709cc52b52279cbcdc" exitCode=0 Nov 03 22:59:05.733870 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:05.733451 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" event={"ID":"e9d87073-2802-41e4-9d83-bc318c44cb87","Type":"ContainerDied","Data":"8dd13e078142ad49d1217f28e8e716bf4d2cea4013065a709cc52b52279cbcdc"} Nov 03 22:59:06.879846 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:06.879820 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:06.973718 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:06.973684 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-openshift-ca\") pod \"e9d87073-2802-41e4-9d83-bc318c44cb87\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " Nov 03 22:59:06.973902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:06.973734 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-xks-ca\") pod \"e9d87073-2802-41e4-9d83-bc318c44cb87\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " Nov 03 22:59:06.973902 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:06.973772 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d87073-2802-41e4-9d83-bc318c44cb87-kube-api-access-nd8m6\" (UniqueName: \"kubernetes.io/projected/e9d87073-2802-41e4-9d83-bc318c44cb87-kube-api-access-nd8m6\") pod \"e9d87073-2802-41e4-9d83-bc318c44cb87\" (UID: \"e9d87073-2802-41e4-9d83-bc318c44cb87\") " Nov 03 22:59:06.974092 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:06.974068 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-openshift-ca" pod "e9d87073-2802-41e4-9d83-bc318c44cb87" (UID: "e9d87073-2802-41e4-9d83-bc318c44cb87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:06.975835 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:06.975811 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-xks-ca" pod "e9d87073-2802-41e4-9d83-bc318c44cb87" (UID: "e9d87073-2802-41e4-9d83-bc318c44cb87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:06.975945 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:06.975864 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d87073-2802-41e4-9d83-bc318c44cb87-kube-api-access-nd8m6" pod "e9d87073-2802-41e4-9d83-bc318c44cb87" (UID: "e9d87073-2802-41e4-9d83-bc318c44cb87"). InnerVolumeSpecName "kube-api-access-nd8m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:07.075473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:07.075383 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd8m6\" (UniqueName: \"kubernetes.io/projected/e9d87073-2802-41e4-9d83-bc318c44cb87-kube-api-access-nd8m6\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:07.075473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:07.075420 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:07.075473 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:07.075434 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d87073-2802-41e4-9d83-bc318c44cb87-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:07.744126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:07.744095 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" Nov 03 22:59:07.744126 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:07.744109 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n" event={"ID":"e9d87073-2802-41e4-9d83-bc318c44cb87","Type":"ContainerDied","Data":"99fdf9bcd852c04e469c4ec67b00a5f6e485f9f4e636217707a95c98bbeb16ae"} Nov 03 22:59:07.744346 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:07.744155 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99fdf9bcd852c04e469c4ec67b00a5f6e485f9f4e636217707a95c98bbeb16ae" Nov 03 22:59:15.065136 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.065098 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr"] Nov 03 22:59:15.065678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.065480 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e9d87073-2802-41e4-9d83-bc318c44cb87" containerName="revoke-keys" Nov 03 22:59:15.065678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.065490 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d87073-2802-41e4-9d83-bc318c44cb87" containerName="revoke-keys" Nov 03 22:59:15.065678 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.065570 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d87073-2802-41e4-9d83-bc318c44cb87" containerName="revoke-keys" Nov 03 22:59:15.068879 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.068864 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:15.147442 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.147410 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpppr\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:15.147626 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.147486 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpppr\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:15.147738 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.147615 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5mq5\" (UniqueName: \"kubernetes.io/projected/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-kube-api-access-m5mq5\") pod \"maas-api-revoke-sub-3d4156f1-xpppr\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:15.172899 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.172869 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:59:15.173042 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.172869 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:59:15.176093 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.176075 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr"] Nov 03 22:59:15.248318 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.248283 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpppr\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:15.248504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.248329 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mq5\" (UniqueName: \"kubernetes.io/projected/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-kube-api-access-m5mq5\") pod \"maas-api-revoke-sub-3d4156f1-xpppr\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:15.248504 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.248377 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpppr\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:15.249033 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.249010 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpppr\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:15.250571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.250548 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpppr\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:15.306408 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.306374 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5mq5\" (UniqueName: \"kubernetes.io/projected/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-kube-api-access-m5mq5\") pod \"maas-api-revoke-sub-3d4156f1-xpppr\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:15.386393 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.386306 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:15.749451 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:59:15.749424 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868cbac1_4039_4fb4_aa7a_158dc6bf7cb0.slice/crio-2f10405a96acd0cd3c0ea964a68a0006e07cf09a7d60ee3fc943c76229c9b8db WatchSource:0}: Error finding container 2f10405a96acd0cd3c0ea964a68a0006e07cf09a7d60ee3fc943c76229c9b8db: Status 404 returned error can't find the container with id 2f10405a96acd0cd3c0ea964a68a0006e07cf09a7d60ee3fc943c76229c9b8db Nov 03 22:59:15.752995 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.752974 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr"] Nov 03 22:59:15.784565 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:15.784535 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" event={"ID":"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0","Type":"ContainerStarted","Data":"2f10405a96acd0cd3c0ea964a68a0006e07cf09a7d60ee3fc943c76229c9b8db"} Nov 03 22:59:16.791254 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:16.791223 4298 generic.go:356] "Generic (PLEG): container finished" podID="868cbac1-4039-4fb4-aa7a-158dc6bf7cb0" containerID="83de7050474f72d507c59b76032c82e64042f5093f0b004cf540251c1d0bfae1" exitCode=0 Nov 03 22:59:16.791706 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:16.791307 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" event={"ID":"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0","Type":"ContainerDied","Data":"83de7050474f72d507c59b76032c82e64042f5093f0b004cf540251c1d0bfae1"} Nov 03 22:59:17.622048 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:59:17.622000 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 22:59:18.118379 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.118355 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:18.205014 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.204981 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-openshift-ca\") pod \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " Nov 03 22:59:18.205222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.205113 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-kube-api-access-m5mq5\" (UniqueName: \"kubernetes.io/projected/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-kube-api-access-m5mq5\") pod \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " Nov 03 22:59:18.205222 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.205146 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-xks-ca\") pod \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\" (UID: \"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0\") " Nov 03 22:59:18.205392 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.205369 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-openshift-ca" pod "868cbac1-4039-4fb4-aa7a-158dc6bf7cb0" (UID: "868cbac1-4039-4fb4-aa7a-158dc6bf7cb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:18.205488 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.205467 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:18.207295 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.207269 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-kube-api-access-m5mq5" pod "868cbac1-4039-4fb4-aa7a-158dc6bf7cb0" (UID: "868cbac1-4039-4fb4-aa7a-158dc6bf7cb0"). InnerVolumeSpecName "kube-api-access-m5mq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:18.207410 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.207333 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-xks-ca" pod "868cbac1-4039-4fb4-aa7a-158dc6bf7cb0" (UID: "868cbac1-4039-4fb4-aa7a-158dc6bf7cb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:18.307208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.307122 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5mq5\" (UniqueName: \"kubernetes.io/projected/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-kube-api-access-m5mq5\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:18.307208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.307174 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:18.812457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.812421 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" Nov 03 22:59:18.812457 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.812443 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr" event={"ID":"868cbac1-4039-4fb4-aa7a-158dc6bf7cb0","Type":"ContainerDied","Data":"2f10405a96acd0cd3c0ea964a68a0006e07cf09a7d60ee3fc943c76229c9b8db"} Nov 03 22:59:18.812713 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:18.812502 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f10405a96acd0cd3c0ea964a68a0006e07cf09a7d60ee3fc943c76229c9b8db" Nov 03 22:59:21.030805 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:21.030774 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:25.667785 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:25.667753 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:27.711516 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:27.711485 4298 scope.go:122] "RemoveContainer" containerID="97c27e090c5212af3f4c343227d529565fde62e85e532dcbc403d3625ee5e469" Nov 03 22:59:27.719751 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:27.719728 4298 scope.go:122] "RemoveContainer" containerID="ac949e45c8f276449be501c59346ef260a30dd1ebdd01a1b372d5f89ccdc40d9" Nov 03 22:59:27.727235 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:27.727214 4298 scope.go:122] "RemoveContainer" containerID="caa52dba89173dfad9c00df80647617993c8e4337841ca67de2960ff5dca43cc" Nov 03 22:59:27.734317 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:27.734295 4298 scope.go:122] "RemoveContainer" containerID="4d0f39741c27f2b0997343d2dc9f2579d60fb1e633113d0b33b44cbf1727c074" Nov 03 22:59:31.329939 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:31.329899 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:33.054384 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.054350 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd"] Nov 03 22:59:33.104914 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.104875 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-25xcd"] Nov 03 22:59:33.107916 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.107893 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr"] Nov 03 22:59:33.110486 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.110462 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpppr"] Nov 03 22:59:33.133820 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.133793 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3042e29b-04a1-4ab2-adaf-20e0b3630395" path="/var/lib/kubelet/pods/3042e29b-04a1-4ab2-adaf-20e0b3630395/volumes" Nov 03 22:59:33.134274 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.134254 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="868cbac1-4039-4fb4-aa7a-158dc6bf7cb0" path="/var/lib/kubelet/pods/868cbac1-4039-4fb4-aa7a-158dc6bf7cb0/volumes" Nov 03 22:59:33.218208 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.218175 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z"] Nov 03 22:59:33.218571 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.218558 4298 cpu_manager.go:414] "containerMap: removing container" podUID="868cbac1-4039-4fb4-aa7a-158dc6bf7cb0" containerName="revoke-keys" Nov 03 22:59:33.218615 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.218572 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868cbac1-4039-4fb4-aa7a-158dc6bf7cb0" containerName="revoke-keys" Nov 03 22:59:33.218674 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.218664 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="868cbac1-4039-4fb4-aa7a-158dc6bf7cb0" containerName="revoke-keys" Nov 03 22:59:33.223128 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.223111 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:33.276429 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.276395 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:59:33.276563 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.276474 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:59:33.279948 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.279928 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z"] Nov 03 22:59:33.347045 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.346962 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tqq7z\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:33.347195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.347074 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tqq7z\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:33.347195 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.347105 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686m8\" (UniqueName: \"kubernetes.io/projected/2c8a377f-7e17-4e62-a581-848e214ec4eb-kube-api-access-686m8\") pod \"maas-api-revoke-sub-3d4156f1-tqq7z\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:33.447827 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.447790 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tqq7z\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:33.447996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.447865 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tqq7z\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:33.447996 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.447886 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686m8\" (UniqueName: \"kubernetes.io/projected/2c8a377f-7e17-4e62-a581-848e214ec4eb-kube-api-access-686m8\") pod \"maas-api-revoke-sub-3d4156f1-tqq7z\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:33.448437 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.448413 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tqq7z\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:33.450182 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.450157 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tqq7z\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:33.505660 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.505618 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686m8\" (UniqueName: \"kubernetes.io/projected/2c8a377f-7e17-4e62-a581-848e214ec4eb-kube-api-access-686m8\") pod \"maas-api-revoke-sub-3d4156f1-tqq7z\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:33.535472 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.535439 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:33.898782 ip-10-0-139-254 kubenswrapper[4298]: W1103 22:59:33.898746 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8a377f_7e17_4e62_a581_848e214ec4eb.slice/crio-611c8d10e1b9c14b16cfac60012d8aa40f8ce82b8ee593071523929567dc2d0b WatchSource:0}: Error finding container 611c8d10e1b9c14b16cfac60012d8aa40f8ce82b8ee593071523929567dc2d0b: Status 404 returned error can't find the container with id 611c8d10e1b9c14b16cfac60012d8aa40f8ce82b8ee593071523929567dc2d0b Nov 03 22:59:33.901252 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:33.901231 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z"] Nov 03 22:59:34.591218 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:34.591138 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:34.888634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:34.888532 4298 generic.go:356] "Generic (PLEG): container finished" podID="2c8a377f-7e17-4e62-a581-848e214ec4eb" containerID="eafb3e0f492e0ec81605ceb508cc05f2315f4fd1630fb8b953c99296341f6807" exitCode=0 Nov 03 22:59:34.888634 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:34.888604 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" event={"ID":"2c8a377f-7e17-4e62-a581-848e214ec4eb","Type":"ContainerDied","Data":"eafb3e0f492e0ec81605ceb508cc05f2315f4fd1630fb8b953c99296341f6807"} Nov 03 22:59:34.888888 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:34.888680 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" event={"ID":"2c8a377f-7e17-4e62-a581-848e214ec4eb","Type":"ContainerStarted","Data":"611c8d10e1b9c14b16cfac60012d8aa40f8ce82b8ee593071523929567dc2d0b"} Nov 03 22:59:36.022668 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.022630 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:36.070822 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.070781 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8a377f-7e17-4e62-a581-848e214ec4eb-kube-api-access-686m8\" (UniqueName: \"kubernetes.io/projected/2c8a377f-7e17-4e62-a581-848e214ec4eb-kube-api-access-686m8\") pod \"2c8a377f-7e17-4e62-a581-848e214ec4eb\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " Nov 03 22:59:36.070975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.070872 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-openshift-ca\") pod \"2c8a377f-7e17-4e62-a581-848e214ec4eb\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " Nov 03 22:59:36.070975 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.070926 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-xks-ca\") pod \"2c8a377f-7e17-4e62-a581-848e214ec4eb\" (UID: \"2c8a377f-7e17-4e62-a581-848e214ec4eb\") " Nov 03 22:59:36.071204 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.071181 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-openshift-ca" pod "2c8a377f-7e17-4e62-a581-848e214ec4eb" (UID: "2c8a377f-7e17-4e62-a581-848e214ec4eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:36.072957 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.072936 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-xks-ca" pod "2c8a377f-7e17-4e62-a581-848e214ec4eb" (UID: "2c8a377f-7e17-4e62-a581-848e214ec4eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:36.073057 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.072967 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8a377f-7e17-4e62-a581-848e214ec4eb-kube-api-access-686m8" pod "2c8a377f-7e17-4e62-a581-848e214ec4eb" (UID: "2c8a377f-7e17-4e62-a581-848e214ec4eb"). InnerVolumeSpecName "kube-api-access-686m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:36.171893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.171792 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-686m8\" (UniqueName: \"kubernetes.io/projected/2c8a377f-7e17-4e62-a581-848e214ec4eb-kube-api-access-686m8\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:36.171893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.171820 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:36.171893 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.171832 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8a377f-7e17-4e62-a581-848e214ec4eb-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:36.899520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.899486 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" Nov 03 22:59:36.899520 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.899513 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z" event={"ID":"2c8a377f-7e17-4e62-a581-848e214ec4eb","Type":"ContainerDied","Data":"611c8d10e1b9c14b16cfac60012d8aa40f8ce82b8ee593071523929567dc2d0b"} Nov 03 22:59:36.899774 ip-10-0-139-254 kubenswrapper[4298]: I1103 22:59:36.899558 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="611c8d10e1b9c14b16cfac60012d8aa40f8ce82b8ee593071523929567dc2d0b" Nov 03 22:59:47.551522 ip-10-0-139-254 kubenswrapper[4298]: E1103 22:59:47.551482 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:00:00.158312 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.158274 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq"] Nov 03 23:00:00.158846 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.158689 4298 cpu_manager.go:414] "containerMap: removing container" podUID="2c8a377f-7e17-4e62-a581-848e214ec4eb" containerName="revoke-keys" Nov 03 23:00:00.158846 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.158701 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8a377f-7e17-4e62-a581-848e214ec4eb" containerName="revoke-keys" Nov 03 23:00:00.158846 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.158792 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8a377f-7e17-4e62-a581-848e214ec4eb" containerName="revoke-keys" Nov 03 23:00:00.161926 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.161906 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" Nov 03 23:00:00.209336 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.209293 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdwj5\" (UniqueName: \"kubernetes.io/projected/3f1601ae-a24f-45a4-80b3-761da07644a5-kube-api-access-zdwj5\") pod \"maas-api-key-cleanup-29895780-znhjq\" (UID: \"3f1601ae-a24f-45a4-80b3-761da07644a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" Nov 03 23:00:00.209556 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.209358 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1601ae-a24f-45a4-80b3-761da07644a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-znhjq\" (UID: \"3f1601ae-a24f-45a4-80b3-761da07644a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" Nov 03 23:00:00.220350 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.220318 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq"] Nov 03 23:00:00.266779 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.266742 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:00:00.310264 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.310225 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdwj5\" (UniqueName: \"kubernetes.io/projected/3f1601ae-a24f-45a4-80b3-761da07644a5-kube-api-access-zdwj5\") pod \"maas-api-key-cleanup-29895780-znhjq\" (UID: \"3f1601ae-a24f-45a4-80b3-761da07644a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" Nov 03 23:00:00.310264 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.310269 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1601ae-a24f-45a4-80b3-761da07644a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-znhjq\" (UID: \"3f1601ae-a24f-45a4-80b3-761da07644a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" Nov 03 23:00:00.310990 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.310972 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1601ae-a24f-45a4-80b3-761da07644a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-znhjq\" (UID: \"3f1601ae-a24f-45a4-80b3-761da07644a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" Nov 03 23:00:00.367362 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.367330 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdwj5\" (UniqueName: \"kubernetes.io/projected/3f1601ae-a24f-45a4-80b3-761da07644a5-kube-api-access-zdwj5\") pod \"maas-api-key-cleanup-29895780-znhjq\" (UID: \"3f1601ae-a24f-45a4-80b3-761da07644a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" Nov 03 23:00:00.475151 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.475118 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" Nov 03 23:00:00.841664 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:00:00.841616 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1601ae_a24f_45a4_80b3_761da07644a5.slice/crio-277b1aa42eedff9defcd98774a44a21e1c346d68ce55e811a9f92d81b28b1ec6 WatchSource:0}: Error finding container 277b1aa42eedff9defcd98774a44a21e1c346d68ce55e811a9f92d81b28b1ec6: Status 404 returned error can't find the container with id 277b1aa42eedff9defcd98774a44a21e1c346d68ce55e811a9f92d81b28b1ec6 Nov 03 23:00:00.846020 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:00.845996 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq"] Nov 03 23:00:01.025000 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:01.024963 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" event={"ID":"3f1601ae-a24f-45a4-80b3-761da07644a5","Type":"ContainerStarted","Data":"277b1aa42eedff9defcd98774a44a21e1c346d68ce55e811a9f92d81b28b1ec6"} Nov 03 23:00:02.103902 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:02.103798 4298 generic.go:356] "Generic (PLEG): container finished" podID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerID="5528549ef713c63eed88bf79086511c4dbb016a06a10c29a7768ab7df6a1cf77" exitCode=60 Nov 03 23:00:02.104421 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:02.103922 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" event={"ID":"3f1601ae-a24f-45a4-80b3-761da07644a5","Type":"ContainerDied","Data":"5528549ef713c63eed88bf79086511c4dbb016a06a10c29a7768ab7df6a1cf77"} Nov 03 23:00:02.104421 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:02.104146 4298 scope.go:122] "RemoveContainer" containerID="5528549ef713c63eed88bf79086511c4dbb016a06a10c29a7768ab7df6a1cf77" Nov 03 23:00:02.594053 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:02.594008 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:03.112298 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:03.112256 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" event={"ID":"3f1601ae-a24f-45a4-80b3-761da07644a5","Type":"ContainerStarted","Data":"cb70d663e69f5c528b6901701c44791059345fd5588b3d24dae0e5ad64ca5383"} Nov 03 23:00:04.119462 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:04.119428 4298 generic.go:356] "Generic (PLEG): container finished" podID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerID="cb70d663e69f5c528b6901701c44791059345fd5588b3d24dae0e5ad64ca5383" exitCode=60 Nov 03 23:00:04.119953 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:04.119511 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" event={"ID":"3f1601ae-a24f-45a4-80b3-761da07644a5","Type":"ContainerDied","Data":"cb70d663e69f5c528b6901701c44791059345fd5588b3d24dae0e5ad64ca5383"} Nov 03 23:00:04.119953 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:04.119567 4298 scope.go:122] "RemoveContainer" containerID="5528549ef713c63eed88bf79086511c4dbb016a06a10c29a7768ab7df6a1cf77" Nov 03 23:00:04.119953 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:04.119787 4298 scope.go:122] "RemoveContainer" containerID="cb70d663e69f5c528b6901701c44791059345fd5588b3d24dae0e5ad64ca5383" Nov 03 23:00:04.120086 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:00:04.120067 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-znhjq_odh-ai-gateway-infra(3f1601ae-a24f-45a4-80b3-761da07644a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" Nov 03 23:00:05.127304 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:05.127274 4298 scope.go:122] "RemoveContainer" containerID="cb70d663e69f5c528b6901701c44791059345fd5588b3d24dae0e5ad64ca5383" Nov 03 23:00:05.127771 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:00:05.127504 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-znhjq_odh-ai-gateway-infra(3f1601ae-a24f-45a4-80b3-761da07644a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" Nov 03 23:00:15.128413 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:15.128376 4298 scope.go:122] "RemoveContainer" containerID="cb70d663e69f5c528b6901701c44791059345fd5588b3d24dae0e5ad64ca5383" Nov 03 23:00:16.182988 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:16.182904 4298 generic.go:356] "Generic (PLEG): container finished" podID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerID="dd9fe001a7d0d95f5a17dcbf134e9a2cb24a7b360423b2abfb67c6e20637e5fe" exitCode=60 Nov 03 23:00:16.182988 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:16.182977 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" event={"ID":"3f1601ae-a24f-45a4-80b3-761da07644a5","Type":"ContainerDied","Data":"dd9fe001a7d0d95f5a17dcbf134e9a2cb24a7b360423b2abfb67c6e20637e5fe"} Nov 03 23:00:16.183470 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:16.183035 4298 scope.go:122] "RemoveContainer" containerID="cb70d663e69f5c528b6901701c44791059345fd5588b3d24dae0e5ad64ca5383" Nov 03 23:00:16.183470 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:16.183350 4298 scope.go:122] "RemoveContainer" containerID="dd9fe001a7d0d95f5a17dcbf134e9a2cb24a7b360423b2abfb67c6e20637e5fe" Nov 03 23:00:16.183615 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:00:16.183592 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-znhjq_odh-ai-gateway-infra(3f1601ae-a24f-45a4-80b3-761da07644a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" Nov 03 23:00:17.307307 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:17.307270 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq"] Nov 03 23:00:17.454304 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:17.454279 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" Nov 03 23:00:17.552212 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:00:17.552170 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:00:17.623017 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:17.622921 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1601ae-a24f-45a4-80b3-761da07644a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1601ae-a24f-45a4-80b3-761da07644a5-maas-api-service-ca\") pod \"3f1601ae-a24f-45a4-80b3-761da07644a5\" (UID: \"3f1601ae-a24f-45a4-80b3-761da07644a5\") " Nov 03 23:00:17.623017 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:17.622986 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1601ae-a24f-45a4-80b3-761da07644a5-kube-api-access-zdwj5\" (UniqueName: \"kubernetes.io/projected/3f1601ae-a24f-45a4-80b3-761da07644a5-kube-api-access-zdwj5\") pod \"3f1601ae-a24f-45a4-80b3-761da07644a5\" (UID: \"3f1601ae-a24f-45a4-80b3-761da07644a5\") " Nov 03 23:00:17.623300 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:17.623278 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1601ae-a24f-45a4-80b3-761da07644a5-maas-api-service-ca" pod "3f1601ae-a24f-45a4-80b3-761da07644a5" (UID: "3f1601ae-a24f-45a4-80b3-761da07644a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:17.625140 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:17.625108 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1601ae-a24f-45a4-80b3-761da07644a5-kube-api-access-zdwj5" pod "3f1601ae-a24f-45a4-80b3-761da07644a5" (UID: "3f1601ae-a24f-45a4-80b3-761da07644a5"). InnerVolumeSpecName "kube-api-access-zdwj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:17.724518 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:17.724481 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1601ae-a24f-45a4-80b3-761da07644a5-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:17.724518 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:17.724515 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdwj5\" (UniqueName: \"kubernetes.io/projected/3f1601ae-a24f-45a4-80b3-761da07644a5-kube-api-access-zdwj5\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:18.195774 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:18.195745 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" Nov 03 23:00:18.195774 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:18.195767 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq" event={"ID":"3f1601ae-a24f-45a4-80b3-761da07644a5","Type":"ContainerDied","Data":"277b1aa42eedff9defcd98774a44a21e1c346d68ce55e811a9f92d81b28b1ec6"} Nov 03 23:00:18.196035 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:18.195819 4298 scope.go:122] "RemoveContainer" containerID="dd9fe001a7d0d95f5a17dcbf134e9a2cb24a7b360423b2abfb67c6e20637e5fe" Nov 03 23:00:18.361740 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:18.361700 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq"] Nov 03 23:00:18.411758 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:18.411719 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-znhjq"] Nov 03 23:00:19.133207 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:19.133164 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" path="/var/lib/kubelet/pods/3f1601ae-a24f-45a4-80b3-761da07644a5/volumes" Nov 03 23:00:21.394678 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.394631 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:21.889995 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.889956 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2"] Nov 03 23:00:21.890355 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.890342 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerName="cleanup" Nov 03 23:00:21.890400 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.890355 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerName="cleanup" Nov 03 23:00:21.890400 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.890364 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerName="cleanup" Nov 03 23:00:21.890400 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.890369 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerName="cleanup" Nov 03 23:00:21.890505 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.890444 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerName="cleanup" Nov 03 23:00:21.890505 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.890456 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerName="cleanup" Nov 03 23:00:21.890505 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.890464 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerName="cleanup" Nov 03 23:00:21.894990 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.894973 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:21.950536 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.950497 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2"] Nov 03 23:00:21.963392 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.963360 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5h6d2\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:21.963562 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.963409 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5h6d2\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:21.963562 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.963435 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldccb\" (UniqueName: \"kubernetes.io/projected/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-kube-api-access-ldccb\") pod \"maas-api-revoke-sub-5a29efb3-5h6d2\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:21.997301 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.997273 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:00:21.997463 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:21.997323 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:00:22.064385 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:22.064344 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldccb\" (UniqueName: \"kubernetes.io/projected/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-kube-api-access-ldccb\") pod \"maas-api-revoke-sub-5a29efb3-5h6d2\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:22.064592 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:22.064445 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5h6d2\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:22.064592 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:22.064488 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5h6d2\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:22.065093 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:22.065074 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5h6d2\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:22.066950 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:22.066922 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5h6d2\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:22.121348 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:22.121310 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldccb\" (UniqueName: \"kubernetes.io/projected/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-kube-api-access-ldccb\") pod \"maas-api-revoke-sub-5a29efb3-5h6d2\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:22.206796 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:22.206763 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:22.570553 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:00:22.570500 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26a73e1_1fe0_4ac3_9c59_814a7de33fec.slice/crio-e1cfa955d201c9251306f73bc59252ec44614a646ce080cfa04cbeb9bea65eba WatchSource:0}: Error finding container e1cfa955d201c9251306f73bc59252ec44614a646ce080cfa04cbeb9bea65eba: Status 404 returned error can't find the container with id e1cfa955d201c9251306f73bc59252ec44614a646ce080cfa04cbeb9bea65eba Nov 03 23:00:22.575388 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:22.575362 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2"] Nov 03 23:00:23.221025 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:23.220990 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" event={"ID":"f26a73e1-1fe0-4ac3-9c59-814a7de33fec","Type":"ContainerStarted","Data":"a4b3e37848b14f372b43b9ddfe07918bc454e11b14172a9a79bd1b90fec3f0bb"} Nov 03 23:00:23.221203 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:23.221041 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" event={"ID":"f26a73e1-1fe0-4ac3-9c59-814a7de33fec","Type":"ContainerStarted","Data":"e1cfa955d201c9251306f73bc59252ec44614a646ce080cfa04cbeb9bea65eba"} Nov 03 23:00:23.336733 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:23.336682 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" podStartSLOduration=2.336667405 podStartE2EDuration="2.336667405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:23.334193409 +0000 UTC m=+3180.314927059" watchObservedRunningTime="2026-11-03 23:00:23.336667405 +0000 UTC m=+3180.317401076" Nov 03 23:00:24.227754 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:24.227712 4298 generic.go:356] "Generic (PLEG): container finished" podID="f26a73e1-1fe0-4ac3-9c59-814a7de33fec" containerID="a4b3e37848b14f372b43b9ddfe07918bc454e11b14172a9a79bd1b90fec3f0bb" exitCode=0 Nov 03 23:00:24.228165 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:24.227795 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" event={"ID":"f26a73e1-1fe0-4ac3-9c59-814a7de33fec","Type":"ContainerDied","Data":"a4b3e37848b14f372b43b9ddfe07918bc454e11b14172a9a79bd1b90fec3f0bb"} Nov 03 23:00:25.600925 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:25.600672 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:25.712741 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:25.712711 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-openshift-ca\") pod \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " Nov 03 23:00:25.712923 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:25.712807 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-kube-api-access-ldccb\" (UniqueName: \"kubernetes.io/projected/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-kube-api-access-ldccb\") pod \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " Nov 03 23:00:25.712923 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:25.712841 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-xks-ca\") pod \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\" (UID: \"f26a73e1-1fe0-4ac3-9c59-814a7de33fec\") " Nov 03 23:00:25.713128 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:25.713103 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-openshift-ca" pod "f26a73e1-1fe0-4ac3-9c59-814a7de33fec" (UID: "f26a73e1-1fe0-4ac3-9c59-814a7de33fec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:25.714895 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:25.714867 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-xks-ca" pod "f26a73e1-1fe0-4ac3-9c59-814a7de33fec" (UID: "f26a73e1-1fe0-4ac3-9c59-814a7de33fec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:25.715915 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:25.715896 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-kube-api-access-ldccb" pod "f26a73e1-1fe0-4ac3-9c59-814a7de33fec" (UID: "f26a73e1-1fe0-4ac3-9c59-814a7de33fec"). InnerVolumeSpecName "kube-api-access-ldccb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:25.814543 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:25.814446 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldccb\" (UniqueName: \"kubernetes.io/projected/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-kube-api-access-ldccb\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:25.814543 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:25.814480 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:25.814543 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:25.814490 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f26a73e1-1fe0-4ac3-9c59-814a7de33fec-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:26.100580 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:26.100481 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:26.281667 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:26.281402 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" event={"ID":"f26a73e1-1fe0-4ac3-9c59-814a7de33fec","Type":"ContainerDied","Data":"e1cfa955d201c9251306f73bc59252ec44614a646ce080cfa04cbeb9bea65eba"} Nov 03 23:00:26.281667 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:26.281484 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1cfa955d201c9251306f73bc59252ec44614a646ce080cfa04cbeb9bea65eba" Nov 03 23:00:26.281667 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:26.281635 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2" Nov 03 23:00:27.762749 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:27.762713 4298 scope.go:122] "RemoveContainer" containerID="2c64cf755e1d07c2f9415f6c485e50087fc9d0fd0b2eaf87cfc5a5fa83c053d2" Nov 03 23:00:31.678203 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:31.678169 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:38.997140 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:38.997060 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z"] Nov 03 23:00:39.047488 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.047443 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tqq7z"] Nov 03 23:00:39.050257 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.050236 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb"] Nov 03 23:00:39.052779 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.052759 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hklqb"] Nov 03 23:00:39.120204 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.120177 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh"] Nov 03 23:00:39.120564 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.120550 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f26a73e1-1fe0-4ac3-9c59-814a7de33fec" containerName="revoke-keys" Nov 03 23:00:39.120608 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.120565 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26a73e1-1fe0-4ac3-9c59-814a7de33fec" containerName="revoke-keys" Nov 03 23:00:39.120608 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.120578 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerName="cleanup" Nov 03 23:00:39.120608 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.120583 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1601ae-a24f-45a4-80b3-761da07644a5" containerName="cleanup" Nov 03 23:00:39.120739 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.120675 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26a73e1-1fe0-4ac3-9c59-814a7de33fec" containerName="revoke-keys" Nov 03 23:00:39.125445 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.125428 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:39.139168 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.139139 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8a377f-7e17-4e62-a581-848e214ec4eb" path="/var/lib/kubelet/pods/2c8a377f-7e17-4e62-a581-848e214ec4eb/volumes" Nov 03 23:00:39.139731 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.139714 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc54ea82-a25c-41d7-b635-f86ed4f7d0fd" path="/var/lib/kubelet/pods/dc54ea82-a25c-41d7-b635-f86ed4f7d0fd/volumes" Nov 03 23:00:39.178691 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.178635 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:00:39.178868 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.178742 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:00:39.181056 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.181030 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh"] Nov 03 23:00:39.242258 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.242229 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sppzh\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:39.242437 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.242285 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sppzh\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:39.242437 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.242396 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77nk2\" (UniqueName: \"kubernetes.io/projected/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-kube-api-access-77nk2\") pod \"maas-api-revoke-sub-3d4156f1-sppzh\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:39.343630 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.343536 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77nk2\" (UniqueName: \"kubernetes.io/projected/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-kube-api-access-77nk2\") pod \"maas-api-revoke-sub-3d4156f1-sppzh\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:39.343630 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.343625 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sppzh\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:39.343906 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.343733 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sppzh\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:39.344368 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.344337 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sppzh\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:39.345942 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.345917 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sppzh\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:39.400916 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.400888 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77nk2\" (UniqueName: \"kubernetes.io/projected/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-kube-api-access-77nk2\") pod \"maas-api-revoke-sub-3d4156f1-sppzh\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:39.442746 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.442719 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:39.804902 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:00:39.804870 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a1341d_c268_4ddc_bfb7_6fcf153834ae.slice/crio-1e40dea8a2cb286a5d6ca5c391b60ae57559c18ed2b2572a47c1fb1bf3809d78 WatchSource:0}: Error finding container 1e40dea8a2cb286a5d6ca5c391b60ae57559c18ed2b2572a47c1fb1bf3809d78: Status 404 returned error can't find the container with id 1e40dea8a2cb286a5d6ca5c391b60ae57559c18ed2b2572a47c1fb1bf3809d78 Nov 03 23:00:39.807704 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:39.807682 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh"] Nov 03 23:00:40.353656 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:40.353554 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" event={"ID":"c7a1341d-c268-4ddc-bfb7-6fcf153834ae","Type":"ContainerStarted","Data":"35273149d98f9f0e5e2a2a190fa80dc0067e5979e17b7b6739728c429cdfb04b"} Nov 03 23:00:40.353656 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:40.353618 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" event={"ID":"c7a1341d-c268-4ddc-bfb7-6fcf153834ae","Type":"ContainerStarted","Data":"1e40dea8a2cb286a5d6ca5c391b60ae57559c18ed2b2572a47c1fb1bf3809d78"} Nov 03 23:00:40.469732 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:40.469683 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" podStartSLOduration=1.469668963 podStartE2EDuration="1.469668963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:40.466622997 +0000 UTC m=+3197.447356650" watchObservedRunningTime="2026-11-03 23:00:40.469668963 +0000 UTC m=+3197.450402617" Nov 03 23:00:41.360905 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:41.360870 4298 generic.go:356] "Generic (PLEG): container finished" podID="c7a1341d-c268-4ddc-bfb7-6fcf153834ae" containerID="35273149d98f9f0e5e2a2a190fa80dc0067e5979e17b7b6739728c429cdfb04b" exitCode=0 Nov 03 23:00:41.361326 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:41.360960 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" event={"ID":"c7a1341d-c268-4ddc-bfb7-6fcf153834ae","Type":"ContainerDied","Data":"35273149d98f9f0e5e2a2a190fa80dc0067e5979e17b7b6739728c429cdfb04b"} Nov 03 23:00:42.493972 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:42.493948 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:42.572827 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:42.572797 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-kube-api-access-77nk2\" (UniqueName: \"kubernetes.io/projected/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-kube-api-access-77nk2\") pod \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " Nov 03 23:00:42.572827 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:42.572833 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-xks-ca\") pod \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " Nov 03 23:00:42.573045 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:42.572908 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-openshift-ca\") pod \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\" (UID: \"c7a1341d-c268-4ddc-bfb7-6fcf153834ae\") " Nov 03 23:00:42.573286 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:42.573260 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-openshift-ca" pod "c7a1341d-c268-4ddc-bfb7-6fcf153834ae" (UID: "c7a1341d-c268-4ddc-bfb7-6fcf153834ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:42.574905 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:42.574880 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-xks-ca" pod "c7a1341d-c268-4ddc-bfb7-6fcf153834ae" (UID: "c7a1341d-c268-4ddc-bfb7-6fcf153834ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:42.574985 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:42.574909 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-kube-api-access-77nk2" pod "c7a1341d-c268-4ddc-bfb7-6fcf153834ae" (UID: "c7a1341d-c268-4ddc-bfb7-6fcf153834ae"). InnerVolumeSpecName "kube-api-access-77nk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:42.674716 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:42.674614 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:42.674716 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:42.674663 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77nk2\" (UniqueName: \"kubernetes.io/projected/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-kube-api-access-77nk2\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:42.674716 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:42.674676 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a1341d-c268-4ddc-bfb7-6fcf153834ae-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:43.372192 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:43.372164 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" Nov 03 23:00:43.372378 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:43.372163 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh" event={"ID":"c7a1341d-c268-4ddc-bfb7-6fcf153834ae","Type":"ContainerDied","Data":"1e40dea8a2cb286a5d6ca5c391b60ae57559c18ed2b2572a47c1fb1bf3809d78"} Nov 03 23:00:43.372378 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:43.372287 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e40dea8a2cb286a5d6ca5c391b60ae57559c18ed2b2572a47c1fb1bf3809d78" Nov 03 23:00:47.552021 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:00:47.551977 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:00:55.226750 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:55.226713 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:55.832877 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:55.832842 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:56.038052 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:56.038002 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:59.960448 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:00:59.960392 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:00.158529 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:00.158490 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:00.350456 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:00.350354 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:03.501797 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.501759 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:03.689867 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.689837 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:03.777934 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.777851 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78"] Nov 03 23:01:03.778298 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.778283 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c7a1341d-c268-4ddc-bfb7-6fcf153834ae" containerName="revoke-keys" Nov 03 23:01:03.778344 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.778302 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a1341d-c268-4ddc-bfb7-6fcf153834ae" containerName="revoke-keys" Nov 03 23:01:03.778380 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.778367 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a1341d-c268-4ddc-bfb7-6fcf153834ae" containerName="revoke-keys" Nov 03 23:01:03.781693 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.781674 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:01:03.835692 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.835655 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-fe3af2\"" Nov 03 23:01:03.864278 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.864244 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nm2h\" (UniqueName: \"kubernetes.io/projected/f28decdf-2620-43dd-b513-7fe283efb795-kube-api-access-7nm2h\") pod \"payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78\" (UID: \"f28decdf-2620-43dd-b513-7fe283efb795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:01:03.864467 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.864350 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28decdf-2620-43dd-b513-7fe283efb795-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78\" (UID: \"f28decdf-2620-43dd-b513-7fe283efb795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:01:03.886401 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.886359 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-2f7d23e4-dockercfg-shdxw\"" Nov 03 23:01:03.889872 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.889844 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78"] Nov 03 23:01:03.965456 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.965413 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nm2h\" (UniqueName: \"kubernetes.io/projected/f28decdf-2620-43dd-b513-7fe283efb795-kube-api-access-7nm2h\") pod \"payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78\" (UID: \"f28decdf-2620-43dd-b513-7fe283efb795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:01:03.965698 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.965490 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28decdf-2620-43dd-b513-7fe283efb795-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78\" (UID: \"f28decdf-2620-43dd-b513-7fe283efb795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:01:03.966230 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:03.966210 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28decdf-2620-43dd-b513-7fe283efb795-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78\" (UID: \"f28decdf-2620-43dd-b513-7fe283efb795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:01:04.022844 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:04.022810 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nm2h\" (UniqueName: \"kubernetes.io/projected/f28decdf-2620-43dd-b513-7fe283efb795-kube-api-access-7nm2h\") pod \"payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78\" (UID: \"f28decdf-2620-43dd-b513-7fe283efb795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:01:04.094676 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:04.094570 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:01:04.459214 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:01:04.459177 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28decdf_2620_43dd_b513_7fe283efb795.slice/crio-11c65643de8c17e96a04a0fed0364c320383ec8166f5a3be233c32bb954d738e WatchSource:0}: Error finding container 11c65643de8c17e96a04a0fed0364c320383ec8166f5a3be233c32bb954d738e: Status 404 returned error can't find the container with id 11c65643de8c17e96a04a0fed0364c320383ec8166f5a3be233c32bb954d738e Nov 03 23:01:04.462099 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:04.462078 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78"] Nov 03 23:01:04.476060 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:04.476029 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" event={"ID":"f28decdf-2620-43dd-b513-7fe283efb795","Type":"ContainerStarted","Data":"11c65643de8c17e96a04a0fed0364c320383ec8166f5a3be233c32bb954d738e"} Nov 03 23:01:05.482724 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:05.482680 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" event={"ID":"f28decdf-2620-43dd-b513-7fe283efb795","Type":"ContainerStarted","Data":"2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe"} Nov 03 23:01:05.483135 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:05.482776 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:01:05.598098 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:05.598039 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" podStartSLOduration=2.5980209800000003 podStartE2EDuration="2.59802098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:05.596401694 +0000 UTC m=+3222.577135347" watchObservedRunningTime="2026-11-03 23:01:05.59802098 +0000 UTC m=+3222.578754636" Nov 03 23:01:06.358636 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:06.358603 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:06.541067 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:06.541030 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:06.721518 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:06.721488 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:11.048500 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:11.048467 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6"] Nov 03 23:01:11.098883 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:11.098854 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ckcg6"] Nov 03 23:01:11.133020 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:11.132996 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7618db-7a07-4634-8ddc-fa2f5f0390d5" path="/var/lib/kubelet/pods/6a7618db-7a07-4634-8ddc-fa2f5f0390d5/volumes" Nov 03 23:01:11.438971 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:11.438873 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:16.435104 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:16.435072 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:16.490253 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:16.490229 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:01:17.551897 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:01:17.551857 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:01:21.331188 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:21.331151 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:21.728097 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:21.728064 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:24.046549 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:24.046513 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z"] Nov 03 23:01:24.096845 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:24.096812 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fb26z"] Nov 03 23:01:24.824417 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:24.824374 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:25.133664 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:25.133570 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb213fc-764a-4f5b-9866-043e798f6a4c" path="/var/lib/kubelet/pods/dbb213fc-764a-4f5b-9866-043e798f6a4c/volumes" Nov 03 23:01:25.406895 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:25.406816 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:25.594695 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:25.594660 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:26.452337 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:26.452302 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:26.661099 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:26.661061 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:27.786123 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:27.786095 4298 scope.go:122] "RemoveContainer" containerID="5c1878043e57bf8923b78bc279706e1f6674c70dc637cb78185529f3b555ea90" Nov 03 23:01:28.998729 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:28.998694 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:31.622396 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:31.622357 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:32.041908 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:32.041874 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:36.557524 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:36.557490 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:41.517109 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:41.517070 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:46.514890 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:46.514856 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:47.552529 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:01:47.552486 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:01:51.527249 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:51.527219 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:56.449703 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:01:56.449634 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:01.339219 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:01.339180 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:02.047591 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:02.047554 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5"] Nov 03 23:02:02.097339 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:02.097303 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vhhd5"] Nov 03 23:02:03.133950 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:03.133861 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849d7411-37d7-44c7-a419-0e0a9f10b59f" path="/var/lib/kubelet/pods/849d7411-37d7-44c7-a419-0e0a9f10b59f/volumes" Nov 03 23:02:06.310609 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:06.310507 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:11.219877 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:11.219843 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:12.844118 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:12.844082 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx"] Nov 03 23:02:12.848015 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:12.847999 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:12.906189 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:12.906162 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c68e0d8-rwgqx\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:12.906359 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:12.906221 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c68e0d8-rwgqx\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:12.906415 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:12.906377 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgn9\" (UniqueName: \"kubernetes.io/projected/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-kube-api-access-rtgn9\") pod \"maas-api-revoke-sub-2c68e0d8-rwgqx\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:12.952005 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:12.951971 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:12.952176 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:12.952036 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:02:12.955175 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:12.955152 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx"] Nov 03 23:02:13.007464 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.007431 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c68e0d8-rwgqx\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:13.007667 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.007516 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgn9\" (UniqueName: \"kubernetes.io/projected/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-kube-api-access-rtgn9\") pod \"maas-api-revoke-sub-2c68e0d8-rwgqx\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:13.007667 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.007557 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c68e0d8-rwgqx\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:13.008119 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.008091 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c68e0d8-rwgqx\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:13.009843 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.009819 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c68e0d8-rwgqx\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:13.046876 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.046845 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn"] Nov 03 23:02:13.049608 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.049583 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m5hdn"] Nov 03 23:02:13.064397 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.064365 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgn9\" (UniqueName: \"kubernetes.io/projected/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-kube-api-access-rtgn9\") pod \"maas-api-revoke-sub-2c68e0d8-rwgqx\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:13.134159 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.134084 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7be1a2a-6348-4909-b059-d0c40df63e99" path="/var/lib/kubelet/pods/f7be1a2a-6348-4909-b059-d0c40df63e99/volumes" Nov 03 23:02:13.160968 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.160944 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:13.528731 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:02:13.528697 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e6b540_32e1_47ef_bee1_fa87b66ed2b8.slice/crio-a5cdb39a2b0298f780221415072478810187b844ebc162c5fbd7fb52d20cd02f WatchSource:0}: Error finding container a5cdb39a2b0298f780221415072478810187b844ebc162c5fbd7fb52d20cd02f: Status 404 returned error can't find the container with id a5cdb39a2b0298f780221415072478810187b844ebc162c5fbd7fb52d20cd02f Nov 03 23:02:13.531064 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.531040 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx"] Nov 03 23:02:13.827019 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:13.826984 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" event={"ID":"22e6b540-32e1-47ef-bee1-fa87b66ed2b8","Type":"ContainerStarted","Data":"a5cdb39a2b0298f780221415072478810187b844ebc162c5fbd7fb52d20cd02f"} Nov 03 23:02:14.836360 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:14.836331 4298 generic.go:356] "Generic (PLEG): container finished" podID="22e6b540-32e1-47ef-bee1-fa87b66ed2b8" containerID="63d4d6c4e6afdc7740f4003904b6e50601e2e261db26ecb0b648b35ae407456e" exitCode=0 Nov 03 23:02:14.836803 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:14.836420 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" event={"ID":"22e6b540-32e1-47ef-bee1-fa87b66ed2b8","Type":"ContainerDied","Data":"63d4d6c4e6afdc7740f4003904b6e50601e2e261db26ecb0b648b35ae407456e"} Nov 03 23:02:15.975271 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:15.975247 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:16.035902 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.035871 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-kube-api-access-rtgn9\" (UniqueName: \"kubernetes.io/projected/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-kube-api-access-rtgn9\") pod \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " Nov 03 23:02:16.036090 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.035943 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-openshift-ca\") pod \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " Nov 03 23:02:16.036090 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.035967 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-xks-ca\") pod \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\" (UID: \"22e6b540-32e1-47ef-bee1-fa87b66ed2b8\") " Nov 03 23:02:16.036383 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.036332 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-openshift-ca" pod "22e6b540-32e1-47ef-bee1-fa87b66ed2b8" (UID: "22e6b540-32e1-47ef-bee1-fa87b66ed2b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:16.038057 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.038035 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-xks-ca" pod "22e6b540-32e1-47ef-bee1-fa87b66ed2b8" (UID: "22e6b540-32e1-47ef-bee1-fa87b66ed2b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:16.038173 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.038051 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-kube-api-access-rtgn9" pod "22e6b540-32e1-47ef-bee1-fa87b66ed2b8" (UID: "22e6b540-32e1-47ef-bee1-fa87b66ed2b8"). InnerVolumeSpecName "kube-api-access-rtgn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:16.137877 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.137802 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtgn9\" (UniqueName: \"kubernetes.io/projected/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-kube-api-access-rtgn9\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:16.137877 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.137831 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:16.137877 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.137842 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e6b540-32e1-47ef-bee1-fa87b66ed2b8-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:16.235711 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.235667 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:16.846926 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.846897 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" Nov 03 23:02:16.847142 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.846938 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx" event={"ID":"22e6b540-32e1-47ef-bee1-fa87b66ed2b8","Type":"ContainerDied","Data":"a5cdb39a2b0298f780221415072478810187b844ebc162c5fbd7fb52d20cd02f"} Nov 03 23:02:16.847142 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:16.846986 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5cdb39a2b0298f780221415072478810187b844ebc162c5fbd7fb52d20cd02f" Nov 03 23:02:17.551495 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:02:17.551449 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:02:21.135093 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:21.135057 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:22.069368 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:22.069330 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:24.538724 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:24.538639 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:25.092006 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.091974 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh"] Nov 03 23:02:25.092459 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.092443 4298 cpu_manager.go:414] "containerMap: removing container" podUID="22e6b540-32e1-47ef-bee1-fa87b66ed2b8" containerName="revoke-keys" Nov 03 23:02:25.092513 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.092461 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e6b540-32e1-47ef-bee1-fa87b66ed2b8" containerName="revoke-keys" Nov 03 23:02:25.092550 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.092535 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e6b540-32e1-47ef-bee1-fa87b66ed2b8" containerName="revoke-keys" Nov 03 23:02:25.095535 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.095520 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:25.200888 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.200854 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:25.200888 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.200854 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:02:25.203864 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.203845 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh"] Nov 03 23:02:25.224625 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.224597 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:25.224741 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.224684 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78g9f\" (UniqueName: \"kubernetes.io/projected/5e7f6572-f518-4a95-94d5-659173457e2e-kube-api-access-78g9f\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:25.224792 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.224761 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:25.325296 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.325240 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78g9f\" (UniqueName: \"kubernetes.io/projected/5e7f6572-f518-4a95-94d5-659173457e2e-kube-api-access-78g9f\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:25.325464 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.325321 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:25.325464 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.325368 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:25.325981 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.325953 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:25.327635 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.327613 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:25.382341 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.382272 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78g9f\" (UniqueName: \"kubernetes.io/projected/5e7f6572-f518-4a95-94d5-659173457e2e-kube-api-access-78g9f\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:25.407476 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.407446 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:25.776237 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:02:25.776204 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7f6572_f518_4a95_94d5_659173457e2e.slice/crio-994aad50935702e1e85f614cd97426a2d3c1b735c872a9d5f2dcb18ec00dd498 WatchSource:0}: Error finding container 994aad50935702e1e85f614cd97426a2d3c1b735c872a9d5f2dcb18ec00dd498: Status 404 returned error can't find the container with id 994aad50935702e1e85f614cd97426a2d3c1b735c872a9d5f2dcb18ec00dd498 Nov 03 23:02:25.778330 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.778308 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh"] Nov 03 23:02:25.891284 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:25.891240 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" event={"ID":"5e7f6572-f518-4a95-94d5-659173457e2e","Type":"ContainerStarted","Data":"994aad50935702e1e85f614cd97426a2d3c1b735c872a9d5f2dcb18ec00dd498"} Nov 03 23:02:26.226605 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:26.226527 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:26.793115 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:26.793073 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:26.897931 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:26.897838 4298 generic.go:356] "Generic (PLEG): container finished" podID="5e7f6572-f518-4a95-94d5-659173457e2e" containerID="c7fcaff78199d68b78d722740582ce3f5e7714ac2ad9e496514eadd263eb3784" exitCode=0 Nov 03 23:02:26.898086 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:26.897923 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" event={"ID":"5e7f6572-f518-4a95-94d5-659173457e2e","Type":"ContainerDied","Data":"c7fcaff78199d68b78d722740582ce3f5e7714ac2ad9e496514eadd263eb3784"} Nov 03 23:02:27.807441 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:27.807413 4298 scope.go:122] "RemoveContainer" containerID="ef28b8208db484b277ea8749f243241aa36cd45128b4c38d2320e3c76955c8aa" Nov 03 23:02:27.816789 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:27.816767 4298 scope.go:122] "RemoveContainer" containerID="26a4f062ddc31b4ea86efbf62a11e310e51161110c9069c7e126dbaf702e3304" Nov 03 23:02:27.824458 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:27.824438 4298 scope.go:122] "RemoveContainer" containerID="db44d14c5165231a66fa1cab5100f033b0871336e4cd6f184ce699f89df000ab" Nov 03 23:02:28.045365 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.045337 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:28.150605 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.150530 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-openshift-ca\") pod \"5e7f6572-f518-4a95-94d5-659173457e2e\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " Nov 03 23:02:28.150605 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.150569 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-xks-ca\") pod \"5e7f6572-f518-4a95-94d5-659173457e2e\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " Nov 03 23:02:28.150605 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.150601 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7f6572-f518-4a95-94d5-659173457e2e-kube-api-access-78g9f\" (UniqueName: \"kubernetes.io/projected/5e7f6572-f518-4a95-94d5-659173457e2e-kube-api-access-78g9f\") pod \"5e7f6572-f518-4a95-94d5-659173457e2e\" (UID: \"5e7f6572-f518-4a95-94d5-659173457e2e\") " Nov 03 23:02:28.150989 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.150964 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-openshift-ca" pod "5e7f6572-f518-4a95-94d5-659173457e2e" (UID: "5e7f6572-f518-4a95-94d5-659173457e2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:28.152762 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.152735 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-xks-ca" pod "5e7f6572-f518-4a95-94d5-659173457e2e" (UID: "5e7f6572-f518-4a95-94d5-659173457e2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:28.152877 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.152737 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7f6572-f518-4a95-94d5-659173457e2e-kube-api-access-78g9f" pod "5e7f6572-f518-4a95-94d5-659173457e2e" (UID: "5e7f6572-f518-4a95-94d5-659173457e2e"). InnerVolumeSpecName "kube-api-access-78g9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:28.251522 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.251480 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:28.251522 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.251512 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7f6572-f518-4a95-94d5-659173457e2e-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:28.251522 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.251524 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78g9f\" (UniqueName: \"kubernetes.io/projected/5e7f6572-f518-4a95-94d5-659173457e2e-kube-api-access-78g9f\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:28.908784 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.908755 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" Nov 03 23:02:28.909186 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.908755 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh" event={"ID":"5e7f6572-f518-4a95-94d5-659173457e2e","Type":"ContainerDied","Data":"994aad50935702e1e85f614cd97426a2d3c1b735c872a9d5f2dcb18ec00dd498"} Nov 03 23:02:28.909186 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:28.908878 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="994aad50935702e1e85f614cd97426a2d3c1b735c872a9d5f2dcb18ec00dd498" Nov 03 23:02:29.607529 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:29.607468 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:31.526363 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:31.526324 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:32.389508 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:32.389480 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:35.864435 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:35.864395 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:36.289446 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:36.289414 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78"] Nov 03 23:02:36.394561 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:36.394531 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:36.572019 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:36.571916 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:40.796697 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:40.796663 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:41.075465 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:41.075385 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:41.289955 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:41.289916 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" podUID="f28decdf-2620-43dd-b513-7fe283efb795" containerName="payload-processing" containerID="cri-o://2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe" gracePeriod=25 Nov 03 23:02:41.393843 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:41.393761 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh"] Nov 03 23:02:41.444224 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:41.444190 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe3af2-h82qh"] Nov 03 23:02:41.877824 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:41.877790 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:43.133328 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:43.133297 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e7f6572-f518-4a95-94d5-659173457e2e" path="/var/lib/kubelet/pods/5e7f6572-f518-4a95-94d5-659173457e2e/volumes" Nov 03 23:02:46.489325 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:46.489285 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" podUID="f28decdf-2620-43dd-b513-7fe283efb795" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:02:47.528724 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:47.528691 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:47.556037 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:02:47.555996 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:02:47.768176 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:47.768138 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:48.022085 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:48.022055 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:48.046463 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:48.046426 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz"] Nov 03 23:02:48.096262 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:48.096224 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4nzjz"] Nov 03 23:02:49.133757 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.133727 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad94ffff-f952-41db-b992-1ee7c97af223" path="/var/lib/kubelet/pods/ad94ffff-f952-41db-b992-1ee7c97af223/volumes" Nov 03 23:02:49.474145 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.474107 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh"] Nov 03 23:02:49.524308 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.524277 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sppzh"] Nov 03 23:02:49.598777 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.598746 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d"] Nov 03 23:02:49.599185 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.599173 4298 cpu_manager.go:414] "containerMap: removing container" podUID="5e7f6572-f518-4a95-94d5-659173457e2e" containerName="revoke-keys" Nov 03 23:02:49.599230 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.599187 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7f6572-f518-4a95-94d5-659173457e2e" containerName="revoke-keys" Nov 03 23:02:49.599267 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.599254 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7f6572-f518-4a95-94d5-659173457e2e" containerName="revoke-keys" Nov 03 23:02:49.603701 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.603681 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:49.656797 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.656759 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:49.656797 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.656797 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:02:49.657259 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.657211 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pzq6d\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:49.657375 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.657281 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gf4b\" (UniqueName: \"kubernetes.io/projected/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-kube-api-access-5gf4b\") pod \"maas-api-revoke-sub-3d4156f1-pzq6d\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:49.657375 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.657332 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pzq6d\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:49.660491 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.660465 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d"] Nov 03 23:02:49.758539 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.758451 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pzq6d\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:49.758539 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.758509 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gf4b\" (UniqueName: \"kubernetes.io/projected/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-kube-api-access-5gf4b\") pod \"maas-api-revoke-sub-3d4156f1-pzq6d\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:49.758539 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.758539 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pzq6d\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:49.759271 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.759252 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pzq6d\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:49.760849 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.760826 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pzq6d\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:49.815531 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.815504 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gf4b\" (UniqueName: \"kubernetes.io/projected/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-kube-api-access-5gf4b\") pod \"maas-api-revoke-sub-3d4156f1-pzq6d\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:49.916255 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:49.916222 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:50.282632 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:02:50.282602 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8459b4b8_0eb5_4c01_8b31_29ade20f0e41.slice/crio-1c4204a7be9f207ba6313f18d1d264de897fc770a545b6113d77d07497de3b4c WatchSource:0}: Error finding container 1c4204a7be9f207ba6313f18d1d264de897fc770a545b6113d77d07497de3b4c: Status 404 returned error can't find the container with id 1c4204a7be9f207ba6313f18d1d264de897fc770a545b6113d77d07497de3b4c Nov 03 23:02:50.285176 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:50.285155 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d"] Nov 03 23:02:51.020366 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:51.020331 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" event={"ID":"8459b4b8-0eb5-4c01-8b31-29ade20f0e41","Type":"ContainerStarted","Data":"204f09335cba7d4ef675b92b174130c8f58e2d1c201a2f82a9e02aff5f6bbc99"} Nov 03 23:02:51.020571 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:51.020382 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" event={"ID":"8459b4b8-0eb5-4c01-8b31-29ade20f0e41","Type":"ContainerStarted","Data":"1c4204a7be9f207ba6313f18d1d264de897fc770a545b6113d77d07497de3b4c"} Nov 03 23:02:51.133429 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:51.133395 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a1341d-c268-4ddc-bfb7-6fcf153834ae" path="/var/lib/kubelet/pods/c7a1341d-c268-4ddc-bfb7-6fcf153834ae/volumes" Nov 03 23:02:51.136051 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:51.136014 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" podStartSLOduration=2.136001525 podStartE2EDuration="2.136001525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:51.133502937 +0000 UTC m=+3328.114236592" watchObservedRunningTime="2026-11-03 23:02:51.136001525 +0000 UTC m=+3328.116735196" Nov 03 23:02:52.026140 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:52.026078 4298 generic.go:356] "Generic (PLEG): container finished" podID="8459b4b8-0eb5-4c01-8b31-29ade20f0e41" containerID="204f09335cba7d4ef675b92b174130c8f58e2d1c201a2f82a9e02aff5f6bbc99" exitCode=0 Nov 03 23:02:52.026539 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:52.026161 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" event={"ID":"8459b4b8-0eb5-4c01-8b31-29ade20f0e41","Type":"ContainerDied","Data":"204f09335cba7d4ef675b92b174130c8f58e2d1c201a2f82a9e02aff5f6bbc99"} Nov 03 23:02:53.166780 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:53.166754 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:53.290395 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:53.290306 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-openshift-ca\") pod \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " Nov 03 23:02:53.290583 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:53.290413 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-xks-ca\") pod \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " Nov 03 23:02:53.290583 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:53.290445 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-kube-api-access-5gf4b\" (UniqueName: \"kubernetes.io/projected/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-kube-api-access-5gf4b\") pod \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\" (UID: \"8459b4b8-0eb5-4c01-8b31-29ade20f0e41\") " Nov 03 23:02:53.290754 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:53.290726 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-openshift-ca" pod "8459b4b8-0eb5-4c01-8b31-29ade20f0e41" (UID: "8459b4b8-0eb5-4c01-8b31-29ade20f0e41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:53.292513 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:53.292490 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-xks-ca" pod "8459b4b8-0eb5-4c01-8b31-29ade20f0e41" (UID: "8459b4b8-0eb5-4c01-8b31-29ade20f0e41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:53.292582 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:53.292509 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-kube-api-access-5gf4b" pod "8459b4b8-0eb5-4c01-8b31-29ade20f0e41" (UID: "8459b4b8-0eb5-4c01-8b31-29ade20f0e41"). InnerVolumeSpecName "kube-api-access-5gf4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:53.392002 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:53.391966 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:53.392002 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:53.392001 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:53.392002 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:53.392009 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gf4b\" (UniqueName: \"kubernetes.io/projected/8459b4b8-0eb5-4c01-8b31-29ade20f0e41-kube-api-access-5gf4b\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:54.037418 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:54.037380 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" Nov 03 23:02:54.037418 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:54.037405 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d" event={"ID":"8459b4b8-0eb5-4c01-8b31-29ade20f0e41","Type":"ContainerDied","Data":"1c4204a7be9f207ba6313f18d1d264de897fc770a545b6113d77d07497de3b4c"} Nov 03 23:02:54.037678 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:54.037493 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c4204a7be9f207ba6313f18d1d264de897fc770a545b6113d77d07497de3b4c" Nov 03 23:02:56.489487 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:02:56.489450 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" podUID="f28decdf-2620-43dd-b513-7fe283efb795" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:03:01.535355 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:01.535326 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:03:01.668500 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:01.668398 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28decdf-2620-43dd-b513-7fe283efb795-kube-api-access-7nm2h\" (UniqueName: \"kubernetes.io/projected/f28decdf-2620-43dd-b513-7fe283efb795-kube-api-access-7nm2h\") pod \"f28decdf-2620-43dd-b513-7fe283efb795\" (UID: \"f28decdf-2620-43dd-b513-7fe283efb795\") " Nov 03 23:03:01.668500 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:01.668461 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28decdf-2620-43dd-b513-7fe283efb795-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28decdf-2620-43dd-b513-7fe283efb795-plugins-config-volume\") pod \"f28decdf-2620-43dd-b513-7fe283efb795\" (UID: \"f28decdf-2620-43dd-b513-7fe283efb795\") " Nov 03 23:03:01.668911 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:01.668880 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28decdf-2620-43dd-b513-7fe283efb795-plugins-config-volume" pod "f28decdf-2620-43dd-b513-7fe283efb795" (UID: "f28decdf-2620-43dd-b513-7fe283efb795"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:01.671029 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:01.670996 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28decdf-2620-43dd-b513-7fe283efb795-kube-api-access-7nm2h" pod "f28decdf-2620-43dd-b513-7fe283efb795" (UID: "f28decdf-2620-43dd-b513-7fe283efb795"). InnerVolumeSpecName "kube-api-access-7nm2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:01.770067 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:01.770032 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nm2h\" (UniqueName: \"kubernetes.io/projected/f28decdf-2620-43dd-b513-7fe283efb795-kube-api-access-7nm2h\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:01.770067 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:01.770063 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28decdf-2620-43dd-b513-7fe283efb795-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:02.082410 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:02.082373 4298 generic.go:356] "Generic (PLEG): container finished" podID="f28decdf-2620-43dd-b513-7fe283efb795" containerID="2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe" exitCode=0 Nov 03 23:03:02.082601 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:02.082438 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" Nov 03 23:03:02.082601 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:02.082447 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" event={"ID":"f28decdf-2620-43dd-b513-7fe283efb795","Type":"ContainerDied","Data":"2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe"} Nov 03 23:03:02.082601 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:02.082531 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78" event={"ID":"f28decdf-2620-43dd-b513-7fe283efb795","Type":"ContainerDied","Data":"11c65643de8c17e96a04a0fed0364c320383ec8166f5a3be233c32bb954d738e"} Nov 03 23:03:02.082601 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:02.082560 4298 scope.go:122] "RemoveContainer" containerID="2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe" Nov 03 23:03:02.091680 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:02.091657 4298 scope.go:122] "RemoveContainer" containerID="2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe" Nov 03 23:03:02.092007 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:03:02.091987 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe\": container with ID starting with 2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe not found: ID does not exist" containerID="2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe" Nov 03 23:03:02.092074 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:02.092021 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe"} err="failed to get container status \"2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe\": rpc error: code = NotFound desc = could not find container \"2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe\": container with ID starting with 2037af7914a0f86033f49540a22dda0cfb436d4605fd79c136c21f5c5a115abe not found: ID does not exist" Nov 03 23:03:02.248463 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:02.248430 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78"] Nov 03 23:03:02.299196 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:02.299156 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe3af2-78fddf65dn4r78"] Nov 03 23:03:03.133777 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:03.133744 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28decdf-2620-43dd-b513-7fe283efb795" path="/var/lib/kubelet/pods/f28decdf-2620-43dd-b513-7fe283efb795/volumes" Nov 03 23:03:06.508188 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:06.508149 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:06.821352 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:06.821256 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:12.355914 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:12.355794 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:16.047290 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:16.047256 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg"] Nov 03 23:03:16.097443 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:16.097411 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bkscg"] Nov 03 23:03:17.134030 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:17.134001 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3158106-e93e-4dfe-84cd-a7a8d7d074f3" path="/var/lib/kubelet/pods/e3158106-e93e-4dfe-84cd-a7a8d7d074f3/volumes" Nov 03 23:03:17.551676 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:03:17.551611 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:03:22.478263 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:22.478234 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:27.149464 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:27.149435 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:27.854051 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:27.854014 4298 scope.go:122] "RemoveContainer" containerID="610150db945598b422f235376f44c294b0489c55efa86b66f5aa8632b98a667c" Nov 03 23:03:27.862370 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:27.862351 4298 scope.go:122] "RemoveContainer" containerID="92026a68a4aede96e6aa7cd79c62874051cb6df52f1de97d4ba9ef7bd20a1dbc" Nov 03 23:03:32.750028 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:32.749991 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:47.048677 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:47.048629 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4"] Nov 03 23:03:47.098911 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:47.098880 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-m6hc4"] Nov 03 23:03:47.134307 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:03:47.134274 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46837d1-667e-49c3-906f-4bdad7520a48" path="/var/lib/kubelet/pods/f46837d1-667e-49c3-906f-4bdad7520a48/volumes" Nov 03 23:03:47.551928 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:03:47.551886 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:04:07.046852 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:07.046815 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n"] Nov 03 23:04:07.096734 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:07.096706 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qwf8n"] Nov 03 23:04:07.136381 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:07.136342 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d87073-2802-41e4-9d83-bc318c44cb87" path="/var/lib/kubelet/pods/e9d87073-2802-41e4-9d83-bc318c44cb87/volumes" Nov 03 23:04:17.551762 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:04:17.551721 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:04:22.779182 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:22.779150 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:27.495421 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:27.495390 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:27.891726 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:27.891694 4298 scope.go:122] "RemoveContainer" containerID="54cfe5b2c090332cd218d491922a7a6a34f4cc3434f6f7f27058ea0044536fef" Nov 03 23:04:27.899950 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:27.899930 4298 scope.go:122] "RemoveContainer" containerID="4f6f1645d5c284dc0bb6d0005df4658816c8492d57341b203c3f200af06ef3a9" Nov 03 23:04:33.100402 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:33.100346 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:34.744790 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:34.744749 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:04:34.745453 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:34.745420 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6rjg7" podUID="da4f3451-f4d3-408e-aad0-67f89b6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:04:34.757770 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:34.757727 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:04:34.758000 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:34.757967 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-6x4mq" podUID="818a85a1-2bd0-4d9e-84ef-951f647247e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:04:34.774755 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:34.774726 4298 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:04:34.774952 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:34.774928 4298 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-tqdf6" podUID="701f8c70-a62b-487e-8008-04258f4d75b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:04:36.469257 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.469220 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l"] Nov 03 23:04:36.469693 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.469664 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f28decdf-2620-43dd-b513-7fe283efb795" containerName="payload-processing" Nov 03 23:04:36.469693 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.469676 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28decdf-2620-43dd-b513-7fe283efb795" containerName="payload-processing" Nov 03 23:04:36.469693 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.469686 4298 cpu_manager.go:414] "containerMap: removing container" podUID="8459b4b8-0eb5-4c01-8b31-29ade20f0e41" containerName="revoke-keys" Nov 03 23:04:36.469693 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.469693 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8459b4b8-0eb5-4c01-8b31-29ade20f0e41" containerName="revoke-keys" Nov 03 23:04:36.469860 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.469783 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28decdf-2620-43dd-b513-7fe283efb795" containerName="payload-processing" Nov 03 23:04:36.469860 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.469792 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="8459b4b8-0eb5-4c01-8b31-29ade20f0e41" containerName="revoke-keys" Nov 03 23:04:36.474699 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.474682 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:36.510073 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.510047 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l6b5l\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:36.510210 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.510087 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l6b5l\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:36.510210 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.510177 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfb2g\" (UniqueName: \"kubernetes.io/projected/eea72b42-7dab-453d-92e7-9b3ee787a948-kube-api-access-tfb2g\") pod \"maas-api-revoke-sub-d576e68b-l6b5l\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:36.531052 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.531024 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l"] Nov 03 23:04:36.577448 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.577418 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:36.577613 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.577517 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:04:36.611614 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.611582 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l6b5l\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:36.611807 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.611623 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l6b5l\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:36.611807 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.611673 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfb2g\" (UniqueName: \"kubernetes.io/projected/eea72b42-7dab-453d-92e7-9b3ee787a948-kube-api-access-tfb2g\") pod \"maas-api-revoke-sub-d576e68b-l6b5l\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:36.612359 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.612337 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l6b5l\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:36.613959 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.613940 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l6b5l\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:36.668603 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.668576 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfb2g\" (UniqueName: \"kubernetes.io/projected/eea72b42-7dab-453d-92e7-9b3ee787a948-kube-api-access-tfb2g\") pod \"maas-api-revoke-sub-d576e68b-l6b5l\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:36.788682 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:36.788588 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:37.156167 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:04:37.156135 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea72b42_7dab_453d_92e7_9b3ee787a948.slice/crio-932b27dd38fcfbee777195943eda448a7349a55a94b709883117f1370b50beb0 WatchSource:0}: Error finding container 932b27dd38fcfbee777195943eda448a7349a55a94b709883117f1370b50beb0: Status 404 returned error can't find the container with id 932b27dd38fcfbee777195943eda448a7349a55a94b709883117f1370b50beb0 Nov 03 23:04:37.158521 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:37.158503 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l"] Nov 03 23:04:37.535181 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:37.535140 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" event={"ID":"eea72b42-7dab-453d-92e7-9b3ee787a948","Type":"ContainerStarted","Data":"79f85bfa5929bdc2f78b0be10b7b28487196cfc384799c5d4abfd7e6c4427d4d"} Nov 03 23:04:37.535569 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:37.535204 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" event={"ID":"eea72b42-7dab-453d-92e7-9b3ee787a948","Type":"ContainerStarted","Data":"932b27dd38fcfbee777195943eda448a7349a55a94b709883117f1370b50beb0"} Nov 03 23:04:37.650931 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:37.650879 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" podStartSLOduration=1.6508672500000001 podStartE2EDuration="1.65086725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:37.647747775 +0000 UTC m=+3434.628481429" watchObservedRunningTime="2026-11-03 23:04:37.65086725 +0000 UTC m=+3434.631600903" Nov 03 23:04:38.541527 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:38.541495 4298 generic.go:356] "Generic (PLEG): container finished" podID="eea72b42-7dab-453d-92e7-9b3ee787a948" containerID="79f85bfa5929bdc2f78b0be10b7b28487196cfc384799c5d4abfd7e6c4427d4d" exitCode=0 Nov 03 23:04:38.541978 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:38.541586 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" event={"ID":"eea72b42-7dab-453d-92e7-9b3ee787a948","Type":"ContainerDied","Data":"79f85bfa5929bdc2f78b0be10b7b28487196cfc384799c5d4abfd7e6c4427d4d"} Nov 03 23:04:39.671274 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:39.671249 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:39.740879 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:39.740845 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-openshift-ca\") pod \"eea72b42-7dab-453d-92e7-9b3ee787a948\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " Nov 03 23:04:39.741049 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:39.740904 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea72b42-7dab-453d-92e7-9b3ee787a948-kube-api-access-tfb2g\" (UniqueName: \"kubernetes.io/projected/eea72b42-7dab-453d-92e7-9b3ee787a948-kube-api-access-tfb2g\") pod \"eea72b42-7dab-453d-92e7-9b3ee787a948\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " Nov 03 23:04:39.741049 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:39.741039 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-xks-ca\") pod \"eea72b42-7dab-453d-92e7-9b3ee787a948\" (UID: \"eea72b42-7dab-453d-92e7-9b3ee787a948\") " Nov 03 23:04:39.741297 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:39.741265 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-openshift-ca" pod "eea72b42-7dab-453d-92e7-9b3ee787a948" (UID: "eea72b42-7dab-453d-92e7-9b3ee787a948"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:39.743139 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:39.743111 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea72b42-7dab-453d-92e7-9b3ee787a948-kube-api-access-tfb2g" pod "eea72b42-7dab-453d-92e7-9b3ee787a948" (UID: "eea72b42-7dab-453d-92e7-9b3ee787a948"). InnerVolumeSpecName "kube-api-access-tfb2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:39.743264 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:39.743245 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-xks-ca" pod "eea72b42-7dab-453d-92e7-9b3ee787a948" (UID: "eea72b42-7dab-453d-92e7-9b3ee787a948"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:39.841865 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:39.841788 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:39.841865 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:39.841815 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea72b42-7dab-453d-92e7-9b3ee787a948-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:39.841865 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:39.841826 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfb2g\" (UniqueName: \"kubernetes.io/projected/eea72b42-7dab-453d-92e7-9b3ee787a948-kube-api-access-tfb2g\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:40.551446 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:40.551422 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" Nov 03 23:04:40.551621 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:40.551461 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l" event={"ID":"eea72b42-7dab-453d-92e7-9b3ee787a948","Type":"ContainerDied","Data":"932b27dd38fcfbee777195943eda448a7349a55a94b709883117f1370b50beb0"} Nov 03 23:04:40.551621 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:04:40.551512 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="932b27dd38fcfbee777195943eda448a7349a55a94b709883117f1370b50beb0" Nov 03 23:04:47.551609 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:04:47.551569 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:05:05.596721 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.596620 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm"] Nov 03 23:05:05.597249 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.597227 4298 cpu_manager.go:414] "containerMap: removing container" podUID="eea72b42-7dab-453d-92e7-9b3ee787a948" containerName="revoke-keys" Nov 03 23:05:05.597322 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.597251 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea72b42-7dab-453d-92e7-9b3ee787a948" containerName="revoke-keys" Nov 03 23:05:05.597389 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.597378 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea72b42-7dab-453d-92e7-9b3ee787a948" containerName="revoke-keys" Nov 03 23:05:05.600798 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.600778 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:05.655326 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.655280 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:05:05.694907 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.694864 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fcmkm\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:05.695087 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.694997 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fcmkm\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:05.695087 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.695030 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vdd\" (UniqueName: \"kubernetes.io/projected/060cda25-96bc-4935-8bbe-05301d943d51-kube-api-access-h5vdd\") pod \"maas-api-revoke-sub-d2bc93bc-fcmkm\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:05.705387 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.705360 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:05:05.708190 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.708168 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm"] Nov 03 23:05:05.796462 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.796423 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fcmkm\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:05.796708 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.796462 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vdd\" (UniqueName: \"kubernetes.io/projected/060cda25-96bc-4935-8bbe-05301d943d51-kube-api-access-h5vdd\") pod \"maas-api-revoke-sub-d2bc93bc-fcmkm\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:05.796708 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.796550 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fcmkm\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:05.797179 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.797153 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fcmkm\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:05.798912 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.798891 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fcmkm\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:05.853706 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.853602 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vdd\" (UniqueName: \"kubernetes.io/projected/060cda25-96bc-4935-8bbe-05301d943d51-kube-api-access-h5vdd\") pod \"maas-api-revoke-sub-d2bc93bc-fcmkm\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:05.917304 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:05.917234 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:06.288472 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:05:06.288417 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060cda25_96bc_4935_8bbe_05301d943d51.slice/crio-5b97a8eba24ceb8c79eb2ec375d96fd5d595984bb1a623651ab98ead57fc507e WatchSource:0}: Error finding container 5b97a8eba24ceb8c79eb2ec375d96fd5d595984bb1a623651ab98ead57fc507e: Status 404 returned error can't find the container with id 5b97a8eba24ceb8c79eb2ec375d96fd5d595984bb1a623651ab98ead57fc507e Nov 03 23:05:06.292986 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:06.292954 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm"] Nov 03 23:05:06.709302 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:06.709266 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" event={"ID":"060cda25-96bc-4935-8bbe-05301d943d51","Type":"ContainerStarted","Data":"a986f56b79b5c76034855d83f479a4575fe0c41db3ba61ea85ebc86ed82bb5cd"} Nov 03 23:05:06.709804 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:06.709316 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" event={"ID":"060cda25-96bc-4935-8bbe-05301d943d51","Type":"ContainerStarted","Data":"5b97a8eba24ceb8c79eb2ec375d96fd5d595984bb1a623651ab98ead57fc507e"} Nov 03 23:05:06.823092 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:06.823039 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" podStartSLOduration=1.823025678 podStartE2EDuration="1.823025678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:06.822746375 +0000 UTC m=+3463.803480028" watchObservedRunningTime="2026-11-03 23:05:06.823025678 +0000 UTC m=+3463.803759330" Nov 03 23:05:07.715711 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:07.715675 4298 generic.go:356] "Generic (PLEG): container finished" podID="060cda25-96bc-4935-8bbe-05301d943d51" containerID="a986f56b79b5c76034855d83f479a4575fe0c41db3ba61ea85ebc86ed82bb5cd" exitCode=0 Nov 03 23:05:07.716189 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:07.715739 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" event={"ID":"060cda25-96bc-4935-8bbe-05301d943d51","Type":"ContainerDied","Data":"a986f56b79b5c76034855d83f479a4575fe0c41db3ba61ea85ebc86ed82bb5cd"} Nov 03 23:05:08.851294 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:08.851274 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:08.924167 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:08.924139 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-xks-ca\") pod \"060cda25-96bc-4935-8bbe-05301d943d51\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " Nov 03 23:05:08.924298 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:08.924201 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060cda25-96bc-4935-8bbe-05301d943d51-kube-api-access-h5vdd\" (UniqueName: \"kubernetes.io/projected/060cda25-96bc-4935-8bbe-05301d943d51-kube-api-access-h5vdd\") pod \"060cda25-96bc-4935-8bbe-05301d943d51\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " Nov 03 23:05:08.924298 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:08.924279 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-openshift-ca\") pod \"060cda25-96bc-4935-8bbe-05301d943d51\" (UID: \"060cda25-96bc-4935-8bbe-05301d943d51\") " Nov 03 23:05:08.924615 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:08.924587 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-openshift-ca" pod "060cda25-96bc-4935-8bbe-05301d943d51" (UID: "060cda25-96bc-4935-8bbe-05301d943d51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:08.926255 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:08.926233 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-xks-ca" pod "060cda25-96bc-4935-8bbe-05301d943d51" (UID: "060cda25-96bc-4935-8bbe-05301d943d51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:08.926351 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:08.926295 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060cda25-96bc-4935-8bbe-05301d943d51-kube-api-access-h5vdd" pod "060cda25-96bc-4935-8bbe-05301d943d51" (UID: "060cda25-96bc-4935-8bbe-05301d943d51"). InnerVolumeSpecName "kube-api-access-h5vdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:09.025399 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:09.025322 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:05:09.025399 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:09.025353 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060cda25-96bc-4935-8bbe-05301d943d51-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:05:09.025399 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:09.025362 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5vdd\" (UniqueName: \"kubernetes.io/projected/060cda25-96bc-4935-8bbe-05301d943d51-kube-api-access-h5vdd\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:05:09.727302 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:09.727259 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" event={"ID":"060cda25-96bc-4935-8bbe-05301d943d51","Type":"ContainerDied","Data":"5b97a8eba24ceb8c79eb2ec375d96fd5d595984bb1a623651ab98ead57fc507e"} Nov 03 23:05:09.727514 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:09.727315 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b97a8eba24ceb8c79eb2ec375d96fd5d595984bb1a623651ab98ead57fc507e" Nov 03 23:05:09.727514 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:09.727272 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm" Nov 03 23:05:12.102632 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:12.102596 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:17.550773 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:05:17.550738 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:05:18.064988 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:18.064947 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:18.689027 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:18.688995 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:19.009795 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:19.009744 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:23.128250 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:23.128214 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:23.309132 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:23.309101 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:23.502689 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:23.502638 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:23.703999 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:23.703967 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:26.046698 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:26.046662 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2"] Nov 03 23:05:26.096760 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:26.096727 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5h6d2"] Nov 03 23:05:26.607514 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:26.607483 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:26.784708 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:26.784671 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:27.134876 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.134803 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26a73e1-1fe0-4ac3-9c59-814a7de33fec" path="/var/lib/kubelet/pods/f26a73e1-1fe0-4ac3-9c59-814a7de33fec/volumes" Nov 03 23:05:27.307465 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.307435 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb"] Nov 03 23:05:27.307836 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.307823 4298 cpu_manager.go:414] "containerMap: removing container" podUID="060cda25-96bc-4935-8bbe-05301d943d51" containerName="revoke-keys" Nov 03 23:05:27.307895 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.307836 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060cda25-96bc-4935-8bbe-05301d943d51" containerName="revoke-keys" Nov 03 23:05:27.307929 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.307923 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="060cda25-96bc-4935-8bbe-05301d943d51" containerName="revoke-keys" Nov 03 23:05:27.310621 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.310604 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:05:27.415503 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.415418 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5b493930-dockercfg-cb77b\"" Nov 03 23:05:27.415503 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.415455 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-28c170\"" Nov 03 23:05:27.418267 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.418245 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb"] Nov 03 23:05:27.498973 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.498938 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb\" (UID: \"ad5f8e91-58e4-4ebe-8241-915e38b47fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:05:27.499158 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.498991 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2sgt\" (UniqueName: \"kubernetes.io/projected/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-kube-api-access-q2sgt\") pod \"payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb\" (UID: \"ad5f8e91-58e4-4ebe-8241-915e38b47fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:05:27.600482 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.600449 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb\" (UID: \"ad5f8e91-58e4-4ebe-8241-915e38b47fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:05:27.600676 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.600511 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2sgt\" (UniqueName: \"kubernetes.io/projected/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-kube-api-access-q2sgt\") pod \"payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb\" (UID: \"ad5f8e91-58e4-4ebe-8241-915e38b47fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:05:27.601121 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.601101 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb\" (UID: \"ad5f8e91-58e4-4ebe-8241-915e38b47fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:05:27.658041 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.658005 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2sgt\" (UniqueName: \"kubernetes.io/projected/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-kube-api-access-q2sgt\") pod \"payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb\" (UID: \"ad5f8e91-58e4-4ebe-8241-915e38b47fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:05:27.848542 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.848509 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:27.922139 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.922109 4298 scope.go:122] "RemoveContainer" containerID="8c14cabdd485ac802ad2c018d091ba7ea99e8d55594211b6ac04f5a25aead3da" Nov 03 23:05:27.922706 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.922682 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:05:27.933475 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.933449 4298 scope.go:122] "RemoveContainer" containerID="8dd13e078142ad49d1217f28e8e716bf4d2cea4013065a709cc52b52279cbcdc" Nov 03 23:05:27.953321 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:27.953295 4298 scope.go:122] "RemoveContainer" containerID="83de7050474f72d507c59b76032c82e64042f5093f0b004cf540251c1d0bfae1" Nov 03 23:05:28.293128 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:05:28.293097 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5f8e91_58e4_4ebe_8241_915e38b47fe4.slice/crio-a050ceafbc883ee22e4ad54343eb20947a3f5b1967626354b8def43b407f0cb9 WatchSource:0}: Error finding container a050ceafbc883ee22e4ad54343eb20947a3f5b1967626354b8def43b407f0cb9: Status 404 returned error can't find the container with id a050ceafbc883ee22e4ad54343eb20947a3f5b1967626354b8def43b407f0cb9 Nov 03 23:05:28.295898 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:28.295880 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb"] Nov 03 23:05:28.827712 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:28.827673 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" event={"ID":"ad5f8e91-58e4-4ebe-8241-915e38b47fe4","Type":"ContainerStarted","Data":"3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7"} Nov 03 23:05:28.827910 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:28.827736 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" event={"ID":"ad5f8e91-58e4-4ebe-8241-915e38b47fe4","Type":"ContainerStarted","Data":"a050ceafbc883ee22e4ad54343eb20947a3f5b1967626354b8def43b407f0cb9"} Nov 03 23:05:28.827910 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:28.827784 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:05:28.943489 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:28.943432 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" podStartSLOduration=1.943417963 podStartE2EDuration="1.943417963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:28.940323642 +0000 UTC m=+3485.921057295" watchObservedRunningTime="2026-11-03 23:05:28.943417963 +0000 UTC m=+3485.924151616" Nov 03 23:05:29.993540 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:29.993506 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:30.179105 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:30.179067 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:30.360996 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:30.360916 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:33.458024 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:33.457987 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:35.129505 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:35.129473 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:39.835427 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:39.835399 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:05:40.149438 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:40.149336 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:45.131311 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:45.131275 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:47.550609 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:05:47.550557 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:05:48.163017 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:48.162979 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:48.775353 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:48.775318 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:48.976985 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:48.976951 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:50.425206 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:50.425176 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:50.747607 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:50.747579 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:52.688935 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:52.688897 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:55.327664 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:05:55.327604 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:00.345731 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:00.345693 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:05.236300 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:05.236224 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:10.248200 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:10.248162 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:15.159632 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:15.159602 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:17.552484 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:06:17.552445 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:06:20.233380 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:20.233330 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:23.481583 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:23.481518 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:25.135697 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:25.135666 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:27.985900 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:27.985867 4298 scope.go:122] "RemoveContainer" containerID="a4b3e37848b14f372b43b9ddfe07918bc454e11b14172a9a79bd1b90fec3f0bb" Nov 03 23:06:27.994286 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:27.994260 4298 scope.go:122] "RemoveContainer" containerID="eafb3e0f492e0ec81605ceb508cc05f2315f4fd1630fb8b953c99296341f6807" Nov 03 23:06:28.200897 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:28.200838 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:30.126342 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:30.126287 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:33.796284 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:33.796205 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:35.008257 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:35.008221 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:37.054051 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.054018 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7"] Nov 03 23:06:37.059291 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.059273 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:37.164883 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.164843 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:37.165072 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.164950 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:06:37.168136 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.168117 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7"] Nov 03 23:06:37.188527 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.188497 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kk2l\" (UniqueName: \"kubernetes.io/projected/797461df-3131-4e61-a78e-4bccb5b1d854-kube-api-access-7kk2l\") pod \"maas-api-revoke-sub-d04ec62f-bxkn7\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:37.188631 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.188561 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d04ec62f-bxkn7\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:37.188709 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.188685 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d04ec62f-bxkn7\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:37.290195 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.290157 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kk2l\" (UniqueName: \"kubernetes.io/projected/797461df-3131-4e61-a78e-4bccb5b1d854-kube-api-access-7kk2l\") pod \"maas-api-revoke-sub-d04ec62f-bxkn7\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:37.290397 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.290257 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d04ec62f-bxkn7\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:37.290397 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.290367 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d04ec62f-bxkn7\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:37.291074 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.291050 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d04ec62f-bxkn7\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:37.292635 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.292615 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d04ec62f-bxkn7\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:37.347423 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.347350 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kk2l\" (UniqueName: \"kubernetes.io/projected/797461df-3131-4e61-a78e-4bccb5b1d854-kube-api-access-7kk2l\") pod \"maas-api-revoke-sub-d04ec62f-bxkn7\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:37.372228 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.372200 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:37.737154 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:06:37.737123 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797461df_3131_4e61_a78e_4bccb5b1d854.slice/crio-36e394bb8763d2f35af3bb4de044f193d813a2ba17056502a34b00af7d92386f WatchSource:0}: Error finding container 36e394bb8763d2f35af3bb4de044f193d813a2ba17056502a34b00af7d92386f: Status 404 returned error can't find the container with id 36e394bb8763d2f35af3bb4de044f193d813a2ba17056502a34b00af7d92386f Nov 03 23:06:37.740729 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:37.740700 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7"] Nov 03 23:06:38.162946 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:38.162902 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" event={"ID":"797461df-3131-4e61-a78e-4bccb5b1d854","Type":"ContainerStarted","Data":"ede102682bb09d5cce7538bd6ee930ae9f1e1e93a28ea377c79e86a4864e300b"} Nov 03 23:06:38.163342 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:38.162959 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" event={"ID":"797461df-3131-4e61-a78e-4bccb5b1d854","Type":"ContainerStarted","Data":"36e394bb8763d2f35af3bb4de044f193d813a2ba17056502a34b00af7d92386f"} Nov 03 23:06:38.276478 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:38.276424 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" podStartSLOduration=1.276410276 podStartE2EDuration="1.276410276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:38.276211025 +0000 UTC m=+3555.256944677" watchObservedRunningTime="2026-11-03 23:06:38.276410276 +0000 UTC m=+3555.257143932" Nov 03 23:06:39.169016 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:39.168980 4298 generic.go:356] "Generic (PLEG): container finished" podID="797461df-3131-4e61-a78e-4bccb5b1d854" containerID="ede102682bb09d5cce7538bd6ee930ae9f1e1e93a28ea377c79e86a4864e300b" exitCode=0 Nov 03 23:06:39.169411 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:39.169041 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" event={"ID":"797461df-3131-4e61-a78e-4bccb5b1d854","Type":"ContainerDied","Data":"ede102682bb09d5cce7538bd6ee930ae9f1e1e93a28ea377c79e86a4864e300b"} Nov 03 23:06:39.949846 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:39.949790 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:40.315794 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:40.315768 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:40.420228 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:40.420190 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-openshift-ca\") pod \"797461df-3131-4e61-a78e-4bccb5b1d854\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " Nov 03 23:06:40.420448 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:40.420279 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797461df-3131-4e61-a78e-4bccb5b1d854-kube-api-access-7kk2l\" (UniqueName: \"kubernetes.io/projected/797461df-3131-4e61-a78e-4bccb5b1d854-kube-api-access-7kk2l\") pod \"797461df-3131-4e61-a78e-4bccb5b1d854\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " Nov 03 23:06:40.420448 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:40.420308 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-xks-ca\") pod \"797461df-3131-4e61-a78e-4bccb5b1d854\" (UID: \"797461df-3131-4e61-a78e-4bccb5b1d854\") " Nov 03 23:06:40.420624 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:40.420598 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-openshift-ca" pod "797461df-3131-4e61-a78e-4bccb5b1d854" (UID: "797461df-3131-4e61-a78e-4bccb5b1d854"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:40.422390 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:40.422365 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-xks-ca" pod "797461df-3131-4e61-a78e-4bccb5b1d854" (UID: "797461df-3131-4e61-a78e-4bccb5b1d854"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:40.422457 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:40.422415 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797461df-3131-4e61-a78e-4bccb5b1d854-kube-api-access-7kk2l" pod "797461df-3131-4e61-a78e-4bccb5b1d854" (UID: "797461df-3131-4e61-a78e-4bccb5b1d854"). InnerVolumeSpecName "kube-api-access-7kk2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:40.521406 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:40.521334 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:40.521406 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:40.521358 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kk2l\" (UniqueName: \"kubernetes.io/projected/797461df-3131-4e61-a78e-4bccb5b1d854-kube-api-access-7kk2l\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:40.521406 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:40.521368 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/797461df-3131-4e61-a78e-4bccb5b1d854-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:41.179201 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:41.179175 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" Nov 03 23:06:41.179382 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:41.179170 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7" event={"ID":"797461df-3131-4e61-a78e-4bccb5b1d854","Type":"ContainerDied","Data":"36e394bb8763d2f35af3bb4de044f193d813a2ba17056502a34b00af7d92386f"} Nov 03 23:06:41.179382 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:41.179291 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e394bb8763d2f35af3bb4de044f193d813a2ba17056502a34b00af7d92386f" Nov 03 23:06:44.859041 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:44.859006 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:47.551578 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:06:47.551539 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:06:48.281344 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.281313 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:48.721556 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.721521 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2"] Nov 03 23:06:48.721956 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.721945 4298 cpu_manager.go:414] "containerMap: removing container" podUID="797461df-3131-4e61-a78e-4bccb5b1d854" containerName="revoke-keys" Nov 03 23:06:48.722000 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.721957 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797461df-3131-4e61-a78e-4bccb5b1d854" containerName="revoke-keys" Nov 03 23:06:48.722042 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.722007 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="797461df-3131-4e61-a78e-4bccb5b1d854" containerName="revoke-keys" Nov 03 23:06:48.725167 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.725150 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:48.783408 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.783374 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2"] Nov 03 23:06:48.800406 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.800371 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:48.800783 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.800434 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2wn\" (UniqueName: \"kubernetes.io/projected/b7b7dc5b-1237-41aa-bfde-6166aca5e689-kube-api-access-xr2wn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:48.800783 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.800546 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:48.830070 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.830039 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:48.830181 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.830073 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:06:48.901179 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.901148 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:48.901340 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.901209 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2wn\" (UniqueName: \"kubernetes.io/projected/b7b7dc5b-1237-41aa-bfde-6166aca5e689-kube-api-access-xr2wn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:48.901340 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.901285 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:48.901985 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.901966 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:48.903546 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.903527 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:48.958001 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:48.957965 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2wn\" (UniqueName: \"kubernetes.io/projected/b7b7dc5b-1237-41aa-bfde-6166aca5e689-kube-api-access-xr2wn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:49.037760 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:49.037679 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:49.402974 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:06:49.402924 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b7dc5b_1237_41aa_bfde_6166aca5e689.slice/crio-69e4f18518c0dba3af315d5d4f1750399e430a97abdbe1c5e3dfb27b3d9e25ad WatchSource:0}: Error finding container 69e4f18518c0dba3af315d5d4f1750399e430a97abdbe1c5e3dfb27b3d9e25ad: Status 404 returned error can't find the container with id 69e4f18518c0dba3af315d5d4f1750399e430a97abdbe1c5e3dfb27b3d9e25ad Nov 03 23:06:49.405749 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:49.405702 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2"] Nov 03 23:06:49.928366 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:49.928282 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:50.223026 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:50.222992 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" event={"ID":"b7b7dc5b-1237-41aa-bfde-6166aca5e689","Type":"ContainerStarted","Data":"3665f2f8ef58183aa21aede3216f76df2087295e076aad067ff641390593f1a1"} Nov 03 23:06:50.223216 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:50.223045 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" event={"ID":"b7b7dc5b-1237-41aa-bfde-6166aca5e689","Type":"ContainerStarted","Data":"69e4f18518c0dba3af315d5d4f1750399e430a97abdbe1c5e3dfb27b3d9e25ad"} Nov 03 23:06:50.339203 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:50.339151 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" podStartSLOduration=2.339135654 podStartE2EDuration="2.339135654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:50.336798153 +0000 UTC m=+3567.317531807" watchObservedRunningTime="2026-11-03 23:06:50.339135654 +0000 UTC m=+3567.319869306" Nov 03 23:06:51.229306 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:51.229272 4298 generic.go:356] "Generic (PLEG): container finished" podID="b7b7dc5b-1237-41aa-bfde-6166aca5e689" containerID="3665f2f8ef58183aa21aede3216f76df2087295e076aad067ff641390593f1a1" exitCode=0 Nov 03 23:06:51.229725 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:51.229352 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" event={"ID":"b7b7dc5b-1237-41aa-bfde-6166aca5e689","Type":"ContainerDied","Data":"3665f2f8ef58183aa21aede3216f76df2087295e076aad067ff641390593f1a1"} Nov 03 23:06:52.368891 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:52.368867 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:52.435655 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:52.435622 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b7dc5b-1237-41aa-bfde-6166aca5e689-kube-api-access-xr2wn\" (UniqueName: \"kubernetes.io/projected/b7b7dc5b-1237-41aa-bfde-6166aca5e689-kube-api-access-xr2wn\") pod \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " Nov 03 23:06:52.435833 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:52.435718 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-xks-ca\") pod \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " Nov 03 23:06:52.435833 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:52.435787 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-openshift-ca\") pod \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\" (UID: \"b7b7dc5b-1237-41aa-bfde-6166aca5e689\") " Nov 03 23:06:52.436149 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:52.436124 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-openshift-ca" pod "b7b7dc5b-1237-41aa-bfde-6166aca5e689" (UID: "b7b7dc5b-1237-41aa-bfde-6166aca5e689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:52.437775 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:52.437746 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-xks-ca" pod "b7b7dc5b-1237-41aa-bfde-6166aca5e689" (UID: "b7b7dc5b-1237-41aa-bfde-6166aca5e689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:52.437891 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:52.437825 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b7dc5b-1237-41aa-bfde-6166aca5e689-kube-api-access-xr2wn" pod "b7b7dc5b-1237-41aa-bfde-6166aca5e689" (UID: "b7b7dc5b-1237-41aa-bfde-6166aca5e689"). InnerVolumeSpecName "kube-api-access-xr2wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:52.536910 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:52.536833 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr2wn\" (UniqueName: \"kubernetes.io/projected/b7b7dc5b-1237-41aa-bfde-6166aca5e689-kube-api-access-xr2wn\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:52.536910 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:52.536861 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:52.536910 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:52.536872 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b7dc5b-1237-41aa-bfde-6166aca5e689-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:53.239910 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:53.239878 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" Nov 03 23:06:53.240089 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:53.239878 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2" event={"ID":"b7b7dc5b-1237-41aa-bfde-6166aca5e689","Type":"ContainerDied","Data":"69e4f18518c0dba3af315d5d4f1750399e430a97abdbe1c5e3dfb27b3d9e25ad"} Nov 03 23:06:53.240089 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:53.240018 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69e4f18518c0dba3af315d5d4f1750399e430a97abdbe1c5e3dfb27b3d9e25ad" Nov 03 23:06:53.296084 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:53.296052 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:55.127494 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:55.127429 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:59.462085 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:59.462002 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:59.826578 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:59.826487 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb"] Nov 03 23:06:59.972133 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:06:59.972104 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:00.156777 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:00.156696 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:04.418122 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:04.418085 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:04.725538 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:04.725508 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:04.827104 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:04.827067 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" podUID="ad5f8e91-58e4-4ebe-8241-915e38b47fe4" containerName="payload-pre-processing" containerID="cri-o://3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7" gracePeriod=25 Nov 03 23:07:05.015749 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.015711 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2"] Nov 03 23:07:05.066008 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.065980 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-28c170-6zhg2"] Nov 03 23:07:05.076074 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.076051 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:07:05.135175 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.135145 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7b7dc5b-1237-41aa-bfde-6166aca5e689" path="/var/lib/kubelet/pods/b7b7dc5b-1237-41aa-bfde-6166aca5e689/volumes" Nov 03 23:07:05.156360 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.156334 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-plugins-config-volume\") pod \"ad5f8e91-58e4-4ebe-8241-915e38b47fe4\" (UID: \"ad5f8e91-58e4-4ebe-8241-915e38b47fe4\") " Nov 03 23:07:05.156519 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.156504 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-kube-api-access-q2sgt\" (UniqueName: \"kubernetes.io/projected/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-kube-api-access-q2sgt\") pod \"ad5f8e91-58e4-4ebe-8241-915e38b47fe4\" (UID: \"ad5f8e91-58e4-4ebe-8241-915e38b47fe4\") " Nov 03 23:07:05.156685 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.156665 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-plugins-config-volume" pod "ad5f8e91-58e4-4ebe-8241-915e38b47fe4" (UID: "ad5f8e91-58e4-4ebe-8241-915e38b47fe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:05.158429 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.158414 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-kube-api-access-q2sgt" pod "ad5f8e91-58e4-4ebe-8241-915e38b47fe4" (UID: "ad5f8e91-58e4-4ebe-8241-915e38b47fe4"). InnerVolumeSpecName "kube-api-access-q2sgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:05.258154 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.258071 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2sgt\" (UniqueName: \"kubernetes.io/projected/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-kube-api-access-q2sgt\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:05.258154 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.258101 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5f8e91-58e4-4ebe-8241-915e38b47fe4-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:05.296102 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.296069 4298 generic.go:356] "Generic (PLEG): container finished" podID="ad5f8e91-58e4-4ebe-8241-915e38b47fe4" containerID="3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7" exitCode=0 Nov 03 23:07:05.296279 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.296116 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" event={"ID":"ad5f8e91-58e4-4ebe-8241-915e38b47fe4","Type":"ContainerDied","Data":"3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7"} Nov 03 23:07:05.296279 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.296120 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" Nov 03 23:07:05.296279 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.296155 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb" event={"ID":"ad5f8e91-58e4-4ebe-8241-915e38b47fe4","Type":"ContainerDied","Data":"a050ceafbc883ee22e4ad54343eb20947a3f5b1967626354b8def43b407f0cb9"} Nov 03 23:07:05.296279 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.296178 4298 scope.go:122] "RemoveContainer" containerID="3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7" Nov 03 23:07:05.305098 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.305081 4298 scope.go:122] "RemoveContainer" containerID="3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7" Nov 03 23:07:05.305335 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:07:05.305318 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7\": container with ID starting with 3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7 not found: ID does not exist" containerID="3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7" Nov 03 23:07:05.305389 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.305346 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7"} err="failed to get container status \"3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7\": rpc error: code = NotFound desc = could not find container \"3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7\": container with ID starting with 3d32056d439462f0ae1c85fbe42ca2e5b7b19a99133549259867ec10643eaea7 not found: ID does not exist" Nov 03 23:07:05.461433 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.461394 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb"] Nov 03 23:07:05.467801 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.467774 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-28c170-664fb5rqmb"] Nov 03 23:07:05.507424 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:05.507400 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:07.133707 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:07.133631 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5f8e91-58e4-4ebe-8241-915e38b47fe4" path="/var/lib/kubelet/pods/ad5f8e91-58e4-4ebe-8241-915e38b47fe4/volumes" Nov 03 23:07:11.160154 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:11.160117 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:11.339505 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:11.339476 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:11.541358 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:11.541327 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:17.051557 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:17.051521 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx"] Nov 03 23:07:17.101616 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:17.101585 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c68e0d8-rwgqx"] Nov 03 23:07:17.133825 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:17.133794 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e6b540-32e1-47ef-bee1-fa87b66ed2b8" path="/var/lib/kubelet/pods/22e6b540-32e1-47ef-bee1-fa87b66ed2b8/volumes" Nov 03 23:07:17.551330 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:07:17.551291 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:07:18.248778 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:18.248743 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:23.845080 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:23.845038 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:28.016068 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:28.016034 4298 scope.go:122] "RemoveContainer" containerID="35273149d98f9f0e5e2a2a190fa80dc0067e5979e17b7b6739728c429cdfb04b" Nov 03 23:07:28.490632 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:28.490598 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:29.852557 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:29.852521 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:30.182850 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:30.182772 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:34.145437 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:34.145402 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:38.894314 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:38.894279 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:44.328357 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:44.328322 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:47.551027 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:07:47.550992 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:07:54.048030 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:54.047994 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d"] Nov 03 23:07:54.097871 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:54.097840 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pzq6d"] Nov 03 23:07:55.133225 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:07:55.133190 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8459b4b8-0eb5-4c01-8b31-29ade20f0e41" path="/var/lib/kubelet/pods/8459b4b8-0eb5-4c01-8b31-29ade20f0e41/volumes" Nov 03 23:08:01.328259 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:01.328225 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:17.552776 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:08:17.552737 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:08:24.259607 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:24.259569 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:28.039287 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:28.039254 4298 scope.go:122] "RemoveContainer" containerID="c7fcaff78199d68b78d722740582ce3f5e7714ac2ad9e496514eadd263eb3784" Nov 03 23:08:28.047633 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:28.047614 4298 scope.go:122] "RemoveContainer" containerID="63d4d6c4e6afdc7740f4003904b6e50601e2e261db26ecb0b648b35ae407456e" Nov 03 23:08:28.879288 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:28.879251 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:34.499889 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:34.499855 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:35.276791 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.276758 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld"] Nov 03 23:08:35.277142 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.277129 4298 cpu_manager.go:414] "containerMap: removing container" podUID="b7b7dc5b-1237-41aa-bfde-6166aca5e689" containerName="revoke-keys" Nov 03 23:08:35.277194 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.277143 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b7dc5b-1237-41aa-bfde-6166aca5e689" containerName="revoke-keys" Nov 03 23:08:35.277194 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.277156 4298 cpu_manager.go:414] "containerMap: removing container" podUID="ad5f8e91-58e4-4ebe-8241-915e38b47fe4" containerName="payload-pre-processing" Nov 03 23:08:35.277194 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.277162 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5f8e91-58e4-4ebe-8241-915e38b47fe4" containerName="payload-pre-processing" Nov 03 23:08:35.277294 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.277244 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b7dc5b-1237-41aa-bfde-6166aca5e689" containerName="revoke-keys" Nov 03 23:08:35.277294 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.277251 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5f8e91-58e4-4ebe-8241-915e38b47fe4" containerName="payload-pre-processing" Nov 03 23:08:35.281799 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.281784 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:35.338121 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.338090 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld"] Nov 03 23:08:35.384989 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.384964 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:08:35.385156 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.385031 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:08:35.473446 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.473409 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lst5m\" (UniqueName: \"kubernetes.io/projected/8a9f8632-8514-45d6-9061-c77b9715daf8-kube-api-access-lst5m\") pod \"maas-api-revoke-sub-452b6d7b-6mjld\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:35.473446 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.473447 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6mjld\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:35.473697 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.473467 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6mjld\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:35.574704 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.574604 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lst5m\" (UniqueName: \"kubernetes.io/projected/8a9f8632-8514-45d6-9061-c77b9715daf8-kube-api-access-lst5m\") pod \"maas-api-revoke-sub-452b6d7b-6mjld\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:35.574704 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.574666 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6mjld\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:35.574704 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.574696 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6mjld\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:35.575398 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.575374 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6mjld\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:35.577004 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.576978 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6mjld\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:35.632209 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.632179 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lst5m\" (UniqueName: \"kubernetes.io/projected/8a9f8632-8514-45d6-9061-c77b9715daf8-kube-api-access-lst5m\") pod \"maas-api-revoke-sub-452b6d7b-6mjld\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:35.895038 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:35.894937 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:36.260890 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:08:36.260841 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9f8632_8514_45d6_9061_c77b9715daf8.slice/crio-6df17614ca179551524fae68146d9d95427b8523fbea8d9aaba82c703e76d44a WatchSource:0}: Error finding container 6df17614ca179551524fae68146d9d95427b8523fbea8d9aaba82c703e76d44a: Status 404 returned error can't find the container with id 6df17614ca179551524fae68146d9d95427b8523fbea8d9aaba82c703e76d44a Nov 03 23:08:36.263611 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:36.263590 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld"] Nov 03 23:08:36.755776 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:36.755742 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" event={"ID":"8a9f8632-8514-45d6-9061-c77b9715daf8","Type":"ContainerStarted","Data":"575398c59534c242d3159692b90e81042f5a0d59e611b35978931809cf431289"} Nov 03 23:08:36.756179 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:36.755818 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" event={"ID":"8a9f8632-8514-45d6-9061-c77b9715daf8","Type":"ContainerStarted","Data":"6df17614ca179551524fae68146d9d95427b8523fbea8d9aaba82c703e76d44a"} Nov 03 23:08:36.871455 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:36.871351 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" podStartSLOduration=1.871337768 podStartE2EDuration="1.871337768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:08:36.869109752 +0000 UTC m=+3673.849843404" watchObservedRunningTime="2026-11-03 23:08:36.871337768 +0000 UTC m=+3673.852071449" Nov 03 23:08:37.763073 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:37.763034 4298 generic.go:356] "Generic (PLEG): container finished" podID="8a9f8632-8514-45d6-9061-c77b9715daf8" containerID="575398c59534c242d3159692b90e81042f5a0d59e611b35978931809cf431289" exitCode=0 Nov 03 23:08:37.763488 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:37.763124 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" event={"ID":"8a9f8632-8514-45d6-9061-c77b9715daf8","Type":"ContainerDied","Data":"575398c59534c242d3159692b90e81042f5a0d59e611b35978931809cf431289"} Nov 03 23:08:39.129847 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.129821 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:39.210305 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.210270 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9f8632-8514-45d6-9061-c77b9715daf8-kube-api-access-lst5m\" (UniqueName: \"kubernetes.io/projected/8a9f8632-8514-45d6-9061-c77b9715daf8-kube-api-access-lst5m\") pod \"8a9f8632-8514-45d6-9061-c77b9715daf8\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " Nov 03 23:08:39.210523 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.210347 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-openshift-ca\") pod \"8a9f8632-8514-45d6-9061-c77b9715daf8\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " Nov 03 23:08:39.210523 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.210406 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-xks-ca\") pod \"8a9f8632-8514-45d6-9061-c77b9715daf8\" (UID: \"8a9f8632-8514-45d6-9061-c77b9715daf8\") " Nov 03 23:08:39.210774 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.210748 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-openshift-ca" pod "8a9f8632-8514-45d6-9061-c77b9715daf8" (UID: "8a9f8632-8514-45d6-9061-c77b9715daf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:39.212468 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.212434 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-xks-ca" pod "8a9f8632-8514-45d6-9061-c77b9715daf8" (UID: "8a9f8632-8514-45d6-9061-c77b9715daf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:39.212576 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.212505 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9f8632-8514-45d6-9061-c77b9715daf8-kube-api-access-lst5m" pod "8a9f8632-8514-45d6-9061-c77b9715daf8" (UID: "8a9f8632-8514-45d6-9061-c77b9715daf8"). InnerVolumeSpecName "kube-api-access-lst5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:39.311196 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.311156 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.311196 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.311185 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lst5m\" (UniqueName: \"kubernetes.io/projected/8a9f8632-8514-45d6-9061-c77b9715daf8-kube-api-access-lst5m\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.311196 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.311195 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9f8632-8514-45d6-9061-c77b9715daf8-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.793126 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.793089 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" event={"ID":"8a9f8632-8514-45d6-9061-c77b9715daf8","Type":"ContainerDied","Data":"6df17614ca179551524fae68146d9d95427b8523fbea8d9aaba82c703e76d44a"} Nov 03 23:08:39.793344 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.793169 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6df17614ca179551524fae68146d9d95427b8523fbea8d9aaba82c703e76d44a" Nov 03 23:08:39.793344 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:39.793117 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld" Nov 03 23:08:47.552106 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:08:47.552059 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:08:48.396460 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:48.396427 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:51.712632 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:51.712600 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:55.154387 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:55.154321 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:59.273132 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.273094 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m"] Nov 03 23:08:59.273566 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.273551 4298 cpu_manager.go:414] "containerMap: removing container" podUID="8a9f8632-8514-45d6-9061-c77b9715daf8" containerName="revoke-keys" Nov 03 23:08:59.273617 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.273568 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9f8632-8514-45d6-9061-c77b9715daf8" containerName="revoke-keys" Nov 03 23:08:59.273670 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.273636 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9f8632-8514-45d6-9061-c77b9715daf8" containerName="revoke-keys" Nov 03 23:08:59.276631 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.276613 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:08:59.330629 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.330590 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8cj\" (UniqueName: \"kubernetes.io/projected/f2dd391a-8c7a-4145-b57c-96242aff2acb-kube-api-access-fk8cj\") pod \"maas-api-revoke-sub-3e31a506-6wh7m\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:08:59.330910 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.330680 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-6wh7m\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:08:59.330910 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.330817 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-6wh7m\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:08:59.331430 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.331407 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:08:59.335091 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.335062 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m"] Nov 03 23:08:59.381583 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.381556 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:08:59.432329 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.432291 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-6wh7m\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:08:59.432572 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.432358 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8cj\" (UniqueName: \"kubernetes.io/projected/f2dd391a-8c7a-4145-b57c-96242aff2acb-kube-api-access-fk8cj\") pod \"maas-api-revoke-sub-3e31a506-6wh7m\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:08:59.432572 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.432420 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-6wh7m\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:08:59.433028 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.433006 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-6wh7m\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:08:59.434750 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.434729 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-6wh7m\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:08:59.489842 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.489802 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8cj\" (UniqueName: \"kubernetes.io/projected/f2dd391a-8c7a-4145-b57c-96242aff2acb-kube-api-access-fk8cj\") pod \"maas-api-revoke-sub-3e31a506-6wh7m\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:08:59.593794 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.593718 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:08:59.963136 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:08:59.963099 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dd391a_8c7a_4145_b57c_96242aff2acb.slice/crio-0300ab57b5cd7d56303615a84b9626b52d1df937e5e58ff3bf914473de65216c WatchSource:0}: Error finding container 0300ab57b5cd7d56303615a84b9626b52d1df937e5e58ff3bf914473de65216c: Status 404 returned error can't find the container with id 0300ab57b5cd7d56303615a84b9626b52d1df937e5e58ff3bf914473de65216c Nov 03 23:08:59.966320 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:08:59.966291 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m"] Nov 03 23:09:00.916276 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:00.916240 4298 generic.go:356] "Generic (PLEG): container finished" podID="f2dd391a-8c7a-4145-b57c-96242aff2acb" containerID="93c8cc6dff0211b7797e62f57c0cb28e30ebabe70510251cfd00c50ee7bca986" exitCode=0 Nov 03 23:09:00.916665 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:00.916308 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" event={"ID":"f2dd391a-8c7a-4145-b57c-96242aff2acb","Type":"ContainerDied","Data":"93c8cc6dff0211b7797e62f57c0cb28e30ebabe70510251cfd00c50ee7bca986"} Nov 03 23:09:00.916665 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:00.916343 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" event={"ID":"f2dd391a-8c7a-4145-b57c-96242aff2acb","Type":"ContainerStarted","Data":"0300ab57b5cd7d56303615a84b9626b52d1df937e5e58ff3bf914473de65216c"} Nov 03 23:09:02.265435 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.265399 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:09:02.404462 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.404421 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-openshift-ca\") pod \"f2dd391a-8c7a-4145-b57c-96242aff2acb\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " Nov 03 23:09:02.404707 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.404486 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2dd391a-8c7a-4145-b57c-96242aff2acb-kube-api-access-fk8cj\" (UniqueName: \"kubernetes.io/projected/f2dd391a-8c7a-4145-b57c-96242aff2acb-kube-api-access-fk8cj\") pod \"f2dd391a-8c7a-4145-b57c-96242aff2acb\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " Nov 03 23:09:02.404707 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.404528 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-xks-ca\") pod \"f2dd391a-8c7a-4145-b57c-96242aff2acb\" (UID: \"f2dd391a-8c7a-4145-b57c-96242aff2acb\") " Nov 03 23:09:02.404978 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.404943 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-openshift-ca" pod "f2dd391a-8c7a-4145-b57c-96242aff2acb" (UID: "f2dd391a-8c7a-4145-b57c-96242aff2acb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:02.406924 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.406883 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dd391a-8c7a-4145-b57c-96242aff2acb-kube-api-access-fk8cj" pod "f2dd391a-8c7a-4145-b57c-96242aff2acb" (UID: "f2dd391a-8c7a-4145-b57c-96242aff2acb"). InnerVolumeSpecName "kube-api-access-fk8cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:02.407420 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.407393 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-xks-ca" pod "f2dd391a-8c7a-4145-b57c-96242aff2acb" (UID: "f2dd391a-8c7a-4145-b57c-96242aff2acb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:02.506376 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.506272 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:02.506376 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.506324 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk8cj\" (UniqueName: \"kubernetes.io/projected/f2dd391a-8c7a-4145-b57c-96242aff2acb-kube-api-access-fk8cj\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:02.506376 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.506340 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dd391a-8c7a-4145-b57c-96242aff2acb-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:02.927040 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.926952 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" Nov 03 23:09:02.927040 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.926985 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m" event={"ID":"f2dd391a-8c7a-4145-b57c-96242aff2acb","Type":"ContainerDied","Data":"0300ab57b5cd7d56303615a84b9626b52d1df937e5e58ff3bf914473de65216c"} Nov 03 23:09:02.927040 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:02.927033 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0300ab57b5cd7d56303615a84b9626b52d1df937e5e58ff3bf914473de65216c" Nov 03 23:09:08.304054 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:08.304023 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:17.553251 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:09:17.553212 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:09:24.547799 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:24.547750 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:28.068548 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.068514 4298 scope.go:122] "RemoveContainer" containerID="204f09335cba7d4ef675b92b174130c8f58e2d1c201a2f82a9e02aff5f6bbc99" Nov 03 23:09:28.847037 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.847003 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4"] Nov 03 23:09:28.847417 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.847401 4298 cpu_manager.go:414] "containerMap: removing container" podUID="f2dd391a-8c7a-4145-b57c-96242aff2acb" containerName="revoke-keys" Nov 03 23:09:28.847417 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.847416 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dd391a-8c7a-4145-b57c-96242aff2acb" containerName="revoke-keys" Nov 03 23:09:28.847573 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.847506 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dd391a-8c7a-4145-b57c-96242aff2acb" containerName="revoke-keys" Nov 03 23:09:28.851945 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.851925 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:28.869958 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.869930 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-258fff3b-pk5z4\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:28.870087 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.870011 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-258fff3b-pk5z4\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:28.870154 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.870132 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsst\" (UniqueName: \"kubernetes.io/projected/e7008ef7-cdec-4378-b3d9-e6f5eca89431-kube-api-access-zfsst\") pod \"maas-api-revoke-sub-258fff3b-pk5z4\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:28.905019 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.904988 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:09:28.955236 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.955204 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:09:28.958170 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.958149 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4"] Nov 03 23:09:28.970778 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.970750 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-258fff3b-pk5z4\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:28.970929 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.970800 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsst\" (UniqueName: \"kubernetes.io/projected/e7008ef7-cdec-4378-b3d9-e6f5eca89431-kube-api-access-zfsst\") pod \"maas-api-revoke-sub-258fff3b-pk5z4\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:28.970929 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.970834 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-258fff3b-pk5z4\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:28.971489 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.971461 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-258fff3b-pk5z4\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:28.973135 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:28.973114 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-258fff3b-pk5z4\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:29.028242 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:29.028214 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsst\" (UniqueName: \"kubernetes.io/projected/e7008ef7-cdec-4378-b3d9-e6f5eca89431-kube-api-access-zfsst\") pod \"maas-api-revoke-sub-258fff3b-pk5z4\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:29.164829 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:29.164746 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:29.208590 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:29.208556 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:29.528417 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:09:29.528379 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7008ef7_cdec_4378_b3d9_e6f5eca89431.slice/crio-7f3396a990b80a290cf628fdf116016ec3a84213dfa8f926c04b628097b1d22c WatchSource:0}: Error finding container 7f3396a990b80a290cf628fdf116016ec3a84213dfa8f926c04b628097b1d22c: Status 404 returned error can't find the container with id 7f3396a990b80a290cf628fdf116016ec3a84213dfa8f926c04b628097b1d22c Nov 03 23:09:29.532623 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:29.532578 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4"] Nov 03 23:09:30.069205 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:30.069170 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" event={"ID":"e7008ef7-cdec-4378-b3d9-e6f5eca89431","Type":"ContainerStarted","Data":"9a5cda43c535574ec754027cf599f5ed8db67f01a8d5ed249011b0328452c62a"} Nov 03 23:09:30.069399 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:30.069226 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" event={"ID":"e7008ef7-cdec-4378-b3d9-e6f5eca89431","Type":"ContainerStarted","Data":"7f3396a990b80a290cf628fdf116016ec3a84213dfa8f926c04b628097b1d22c"} Nov 03 23:09:30.183276 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:30.183213 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" podStartSLOduration=2.183194826 podStartE2EDuration="2.183194826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:30.18215088 +0000 UTC m=+3727.162884532" watchObservedRunningTime="2026-11-03 23:09:30.183194826 +0000 UTC m=+3727.163928481" Nov 03 23:09:31.075112 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:31.075079 4298 generic.go:356] "Generic (PLEG): container finished" podID="e7008ef7-cdec-4378-b3d9-e6f5eca89431" containerID="9a5cda43c535574ec754027cf599f5ed8db67f01a8d5ed249011b0328452c62a" exitCode=0 Nov 03 23:09:31.075293 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:31.075168 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" event={"ID":"e7008ef7-cdec-4378-b3d9-e6f5eca89431","Type":"ContainerDied","Data":"9a5cda43c535574ec754027cf599f5ed8db67f01a8d5ed249011b0328452c62a"} Nov 03 23:09:32.207687 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:32.207662 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:32.304347 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:32.304313 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7008ef7-cdec-4378-b3d9-e6f5eca89431-kube-api-access-zfsst\" (UniqueName: \"kubernetes.io/projected/e7008ef7-cdec-4378-b3d9-e6f5eca89431-kube-api-access-zfsst\") pod \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " Nov 03 23:09:32.304532 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:32.304477 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-openshift-ca\") pod \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " Nov 03 23:09:32.304532 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:32.304520 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-xks-ca\") pod \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\" (UID: \"e7008ef7-cdec-4378-b3d9-e6f5eca89431\") " Nov 03 23:09:32.304921 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:32.304898 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-openshift-ca" pod "e7008ef7-cdec-4378-b3d9-e6f5eca89431" (UID: "e7008ef7-cdec-4378-b3d9-e6f5eca89431"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:32.312768 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:32.312732 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-xks-ca" pod "e7008ef7-cdec-4378-b3d9-e6f5eca89431" (UID: "e7008ef7-cdec-4378-b3d9-e6f5eca89431"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:32.312947 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:32.312765 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7008ef7-cdec-4378-b3d9-e6f5eca89431-kube-api-access-zfsst" pod "e7008ef7-cdec-4378-b3d9-e6f5eca89431" (UID: "e7008ef7-cdec-4378-b3d9-e6f5eca89431"). InnerVolumeSpecName "kube-api-access-zfsst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:32.406057 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:32.405975 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:32.406057 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:32.406010 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7008ef7-cdec-4378-b3d9-e6f5eca89431-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:32.406057 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:32.406020 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfsst\" (UniqueName: \"kubernetes.io/projected/e7008ef7-cdec-4378-b3d9-e6f5eca89431-kube-api-access-zfsst\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:33.085348 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:33.085278 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" Nov 03 23:09:33.085348 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:33.085303 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4" event={"ID":"e7008ef7-cdec-4378-b3d9-e6f5eca89431","Type":"ContainerDied","Data":"7f3396a990b80a290cf628fdf116016ec3a84213dfa8f926c04b628097b1d22c"} Nov 03 23:09:33.085550 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:33.085374 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f3396a990b80a290cf628fdf116016ec3a84213dfa8f926c04b628097b1d22c" Nov 03 23:09:34.851099 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:34.851055 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:40.048321 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:40.048287 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l"] Nov 03 23:09:40.099040 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:40.099004 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l6b5l"] Nov 03 23:09:41.133086 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:41.133051 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea72b42-7dab-453d-92e7-9b3ee787a948" path="/var/lib/kubelet/pods/eea72b42-7dab-453d-92e7-9b3ee787a948/volumes" Nov 03 23:09:47.552865 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:09:47.552823 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:09:48.999635 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:48.999601 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:49.461352 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.461318 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk"] Nov 03 23:09:49.461978 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.461957 4298 cpu_manager.go:414] "containerMap: removing container" podUID="e7008ef7-cdec-4378-b3d9-e6f5eca89431" containerName="revoke-keys" Nov 03 23:09:49.462035 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.461981 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7008ef7-cdec-4378-b3d9-e6f5eca89431" containerName="revoke-keys" Nov 03 23:09:49.462078 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.462062 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7008ef7-cdec-4378-b3d9-e6f5eca89431" containerName="revoke-keys" Nov 03 23:09:49.465913 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.465897 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.539302 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.539264 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f18d5a31-openshift-default-dockercfg-r2wpf\"" Nov 03 23:09:49.569960 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.567720 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-credential-socket\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.569960 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.567815 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-token\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.569960 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.567870 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-envoy\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.569960 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.567913 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-certs\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.569960 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.567940 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpsjg\" (UniqueName: \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-kube-api-access-qpsjg\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.569960 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.568113 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-data\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.569960 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.568157 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-socket\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.569960 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.568222 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-podinfo\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.569960 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.568255 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istiod-ca-cert\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.572678 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.572629 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk"] Nov 03 23:09:49.597968 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.597938 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:49.669150 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669119 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-credential-socket\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.669150 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669161 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-token\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.669451 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669188 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-envoy\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.669451 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669230 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-certs\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.669451 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669250 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpsjg\" (UniqueName: \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-kube-api-access-qpsjg\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.669451 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669302 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-data\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.669451 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669327 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-socket\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.669451 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669358 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-podinfo\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.669451 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669388 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istiod-ca-cert\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.669787 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669656 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-credential-socket\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.669787 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669715 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-certs\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.669873 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.669845 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-socket\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.670104 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.670078 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-data\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.670191 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.670153 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istiod-ca-cert\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.671531 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.671506 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-envoy\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.672377 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.672354 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-podinfo\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.726817 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.726738 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-token\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.726993 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.726857 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpsjg\" (UniqueName: \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-kube-api-access-qpsjg\") pod \"e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.777771 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.777737 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:49.784159 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:49.784131 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:50.192833 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:09:50.192796 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc869bbe6_e482_4635_ba65_eb7d42d8e0ea.slice/crio-67f6614c7127655800d2733decf68d435b5dba6a4e583f8da87ef16630cdff15 WatchSource:0}: Error finding container 67f6614c7127655800d2733decf68d435b5dba6a4e583f8da87ef16630cdff15: Status 404 returned error can't find the container with id 67f6614c7127655800d2733decf68d435b5dba6a4e583f8da87ef16630cdff15 Nov 03 23:09:50.195145 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:50.195111 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 23:09:50.195244 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:50.195180 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 23:09:50.195244 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:50.195212 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 23:09:50.196063 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:50.196038 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk"] Nov 03 23:09:51.177522 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:51.177487 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" event={"ID":"c869bbe6-e482-4635-ba65-eb7d42d8e0ea","Type":"ContainerStarted","Data":"296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e"} Nov 03 23:09:51.177736 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:51.177564 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" event={"ID":"c869bbe6-e482-4635-ba65-eb7d42d8e0ea","Type":"ContainerStarted","Data":"67f6614c7127655800d2733decf68d435b5dba6a4e583f8da87ef16630cdff15"} Nov 03 23:09:51.348239 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:51.348180 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" podStartSLOduration=2.348164673 podStartE2EDuration="2.348164673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:51.345760819 +0000 UTC m=+3748.326494473" watchObservedRunningTime="2026-11-03 23:09:51.348164673 +0000 UTC m=+3748.328898327" Nov 03 23:09:51.777968 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:51.777925 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:51.782661 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:51.782615 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:52.182110 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:52.182031 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:52.183109 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:52.183092 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:09:53.514489 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:53.514453 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:53.698823 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:53.698776 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:53.883954 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:53.883863 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:56.898479 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:56.898444 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:57.079902 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:57.079872 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:58.179969 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:58.179933 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:58.358509 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:58.358470 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:58.544774 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:09:58.544740 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:03.424233 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:03.424198 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:08.417848 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:08.417814 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:09.051571 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:09.051540 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm"] Nov 03 23:10:09.098539 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:09.098504 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fcmkm"] Nov 03 23:10:09.152925 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:09.152885 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060cda25-96bc-4935-8bbe-05301d943d51" path="/var/lib/kubelet/pods/060cda25-96bc-4935-8bbe-05301d943d51/volumes" Nov 03 23:10:13.342002 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:13.341954 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:15.294492 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:15.294418 4298 generic.go:356] "Generic (PLEG): container finished" podID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerID="296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e" exitCode=0 Nov 03 23:10:15.294928 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:15.294490 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" event={"ID":"c869bbe6-e482-4635-ba65-eb7d42d8e0ea","Type":"ContainerDied","Data":"296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e"} Nov 03 23:10:15.294928 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:15.294875 4298 scope.go:122] "RemoveContainer" containerID="296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e" Nov 03 23:10:15.296041 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:15.296016 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 23:10:15.296113 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:15.296081 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 23:10:15.296154 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:15.296113 4298 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 03 23:10:16.301059 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:16.301030 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" event={"ID":"c869bbe6-e482-4635-ba65-eb7d42d8e0ea","Type":"ContainerStarted","Data":"4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac"} Nov 03 23:10:16.778019 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:16.777982 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:10:16.778307 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:16.778285 4298 patch_prober.go:29] interesting pod/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 23:10:16.778383 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:16.778330 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 23:10:17.574483 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:10:17.574421 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:10:17.778345 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:17.778179 4298 patch_prober.go:29] interesting pod/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 23:10:17.778345 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:17.778247 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 23:10:18.546381 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:18.546293 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:18.778224 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:18.778182 4298 patch_prober.go:29] interesting pod/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 23:10:18.778635 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:18.778244 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 23:10:19.778125 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:19.778089 4298 patch_prober.go:29] interesting pod/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 23:10:19.778315 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:19.778141 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 23:10:20.778739 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:20.778702 4298 patch_prober.go:29] interesting pod/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 23:10:20.779155 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:20.778768 4298 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 23:10:21.782482 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:21.782448 4298 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:10:21.782977 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:21.782766 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:10:21.783431 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:21.783414 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:10:23.722234 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:23.722122 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:24.885909 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:24.885876 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:28.741507 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:28.741472 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:29.540605 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:29.540575 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:30.044190 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.044152 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp"] Nov 03 23:10:30.047818 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.047801 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:30.103359 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.103327 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:10:30.153393 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.153361 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sf592\"" Nov 03 23:10:30.156273 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.156252 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp"] Nov 03 23:10:30.190025 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.189986 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:30.190025 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.190025 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:30.190237 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.190053 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvggg\" (UniqueName: \"kubernetes.io/projected/3d5646d6-a2c9-4e68-81da-ac2ce456127f-kube-api-access-pvggg\") pod \"maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:30.291543 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.291507 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:30.291543 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.291547 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:30.291810 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.291576 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvggg\" (UniqueName: \"kubernetes.io/projected/3d5646d6-a2c9-4e68-81da-ac2ce456127f-kube-api-access-pvggg\") pod \"maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:30.292148 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.292128 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:30.293902 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.293882 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:30.348593 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.348519 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvggg\" (UniqueName: \"kubernetes.io/projected/3d5646d6-a2c9-4e68-81da-ac2ce456127f-kube-api-access-pvggg\") pod \"maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:30.360117 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.360097 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:30.724774 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:10:30.724745 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5646d6_a2c9_4e68_81da_ac2ce456127f.slice/crio-cec03897b8fd3512e1b8824a458529ab768602d06143d84bf0fac816642c3df2 WatchSource:0}: Error finding container cec03897b8fd3512e1b8824a458529ab768602d06143d84bf0fac816642c3df2: Status 404 returned error can't find the container with id cec03897b8fd3512e1b8824a458529ab768602d06143d84bf0fac816642c3df2 Nov 03 23:10:30.727181 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:30.727154 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp"] Nov 03 23:10:31.372556 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:31.372461 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" event={"ID":"3d5646d6-a2c9-4e68-81da-ac2ce456127f","Type":"ContainerStarted","Data":"c147f63c83b77b1b660c035356996ca45e55f0f201f8c0747909615e239a5c32"} Nov 03 23:10:31.372556 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:31.372512 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" event={"ID":"3d5646d6-a2c9-4e68-81da-ac2ce456127f","Type":"ContainerStarted","Data":"cec03897b8fd3512e1b8824a458529ab768602d06143d84bf0fac816642c3df2"} Nov 03 23:10:31.487703 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:31.487627 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" podStartSLOduration=1.487612357 podStartE2EDuration="1.487612357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:10:31.485501924 +0000 UTC m=+3788.466235577" watchObservedRunningTime="2026-11-03 23:10:31.487612357 +0000 UTC m=+3788.468346010" Nov 03 23:10:32.378303 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:32.378269 4298 generic.go:356] "Generic (PLEG): container finished" podID="3d5646d6-a2c9-4e68-81da-ac2ce456127f" containerID="c147f63c83b77b1b660c035356996ca45e55f0f201f8c0747909615e239a5c32" exitCode=0 Nov 03 23:10:32.378795 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:32.378316 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" event={"ID":"3d5646d6-a2c9-4e68-81da-ac2ce456127f","Type":"ContainerDied","Data":"c147f63c83b77b1b660c035356996ca45e55f0f201f8c0747909615e239a5c32"} Nov 03 23:10:33.512710 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:33.512667 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:33.623214 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:33.623185 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:33.623409 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:33.623228 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-openshift-ca\") pod \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " Nov 03 23:10:33.623409 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:33.623296 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5646d6-a2c9-4e68-81da-ac2ce456127f-kube-api-access-pvggg\" (UniqueName: \"kubernetes.io/projected/3d5646d6-a2c9-4e68-81da-ac2ce456127f-kube-api-access-pvggg\") pod \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " Nov 03 23:10:33.623409 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:33.623367 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-xks-ca\") pod \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\" (UID: \"3d5646d6-a2c9-4e68-81da-ac2ce456127f\") " Nov 03 23:10:33.623619 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:33.623592 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-openshift-ca" pod "3d5646d6-a2c9-4e68-81da-ac2ce456127f" (UID: "3d5646d6-a2c9-4e68-81da-ac2ce456127f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:33.625457 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:33.625428 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-xks-ca" pod "3d5646d6-a2c9-4e68-81da-ac2ce456127f" (UID: "3d5646d6-a2c9-4e68-81da-ac2ce456127f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:33.625584 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:33.625512 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5646d6-a2c9-4e68-81da-ac2ce456127f-kube-api-access-pvggg" pod "3d5646d6-a2c9-4e68-81da-ac2ce456127f" (UID: "3d5646d6-a2c9-4e68-81da-ac2ce456127f"). InnerVolumeSpecName "kube-api-access-pvggg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:33.724286 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:33.724251 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:33.724286 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:33.724280 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvggg\" (UniqueName: \"kubernetes.io/projected/3d5646d6-a2c9-4e68-81da-ac2ce456127f-kube-api-access-pvggg\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:33.724286 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:33.724290 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5646d6-a2c9-4e68-81da-ac2ce456127f-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:34.388811 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:34.388776 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" event={"ID":"3d5646d6-a2c9-4e68-81da-ac2ce456127f","Type":"ContainerDied","Data":"cec03897b8fd3512e1b8824a458529ab768602d06143d84bf0fac816642c3df2"} Nov 03 23:10:34.388994 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:34.388815 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp" Nov 03 23:10:34.388994 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:34.388826 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cec03897b8fd3512e1b8824a458529ab768602d06143d84bf0fac816642c3df2" Nov 03 23:10:35.193822 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:35.193790 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:38.615791 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:38.615757 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:39.950273 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:39.950240 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:40.126319 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:40.126284 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:41.973873 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:41.973821 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:42.151624 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:42.151593 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:43.518809 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:43.518773 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:47.552804 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:10:47.552766 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:10:47.828865 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:47.828787 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:49.400726 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:49.400689 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:49.684345 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:49.684261 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:49.875713 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:49.875665 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp"] Nov 03 23:10:49.926076 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:49.926044 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f18d5a31-wvzfp"] Nov 03 23:10:51.133244 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:10:51.133204 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5646d6-a2c9-4e68-81da-ac2ce456127f" path="/var/lib/kubelet/pods/3d5646d6-a2c9-4e68-81da-ac2ce456127f/volumes" Nov 03 23:11:11.128039 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:11.127944 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk"] Nov 03 23:11:11.128500 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:11.128250 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" containerID="cri-o://4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac" gracePeriod=30 Nov 03 23:11:11.173613 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:11.173574 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:16.376379 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.376344 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:11:16.542564 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.542530 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-certs\") pod \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " Nov 03 23:11:16.542564 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.542568 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-token\" (UniqueName: \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-token\") pod \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " Nov 03 23:11:16.542842 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.542596 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-data\") pod \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " Nov 03 23:11:16.542842 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.542704 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-credential-socket\") pod \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " Nov 03 23:11:16.542842 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.542740 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-podinfo\") pod \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " Nov 03 23:11:16.542842 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.542771 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istiod-ca-cert\") pod \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " Nov 03 23:11:16.542842 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.542802 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-socket\") pod \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " Nov 03 23:11:16.543131 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.542842 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-kube-api-access-qpsjg\" (UniqueName: \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-kube-api-access-qpsjg\") pod \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " Nov 03 23:11:16.543131 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.542875 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-envoy\") pod \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\" (UID: \"c869bbe6-e482-4635-ba65-eb7d42d8e0ea\") " Nov 03 23:11:16.543131 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.542879 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-data" pod "c869bbe6-e482-4635-ba65-eb7d42d8e0ea" (UID: "c869bbe6-e482-4635-ba65-eb7d42d8e0ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:16.543131 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.542907 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-certs" pod "c869bbe6-e482-4635-ba65-eb7d42d8e0ea" (UID: "c869bbe6-e482-4635-ba65-eb7d42d8e0ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:16.543131 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.543091 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-credential-socket" pod "c869bbe6-e482-4635-ba65-eb7d42d8e0ea" (UID: "c869bbe6-e482-4635-ba65-eb7d42d8e0ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:16.543131 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.543127 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istiod-ca-cert" pod "c869bbe6-e482-4635-ba65-eb7d42d8e0ea" (UID: "c869bbe6-e482-4635-ba65-eb7d42d8e0ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:16.543474 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.543225 4298 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-credential-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:16.543474 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.543242 4298 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istiod-ca-cert\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:16.543474 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.543255 4298 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-certs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:16.543474 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.543267 4298 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-data\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:16.543474 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.543404 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-socket" pod "c869bbe6-e482-4635-ba65-eb7d42d8e0ea" (UID: "c869bbe6-e482-4635-ba65-eb7d42d8e0ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:16.545351 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.545324 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-envoy" pod "c869bbe6-e482-4635-ba65-eb7d42d8e0ea" (UID: "c869bbe6-e482-4635-ba65-eb7d42d8e0ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:16.545491 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.545379 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-kube-api-access-qpsjg" pod "c869bbe6-e482-4635-ba65-eb7d42d8e0ea" (UID: "c869bbe6-e482-4635-ba65-eb7d42d8e0ea"). InnerVolumeSpecName "kube-api-access-qpsjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:16.545606 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.545587 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-podinfo" pod "c869bbe6-e482-4635-ba65-eb7d42d8e0ea" (UID: "c869bbe6-e482-4635-ba65-eb7d42d8e0ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:11:16.545689 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.545632 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-token" pod "c869bbe6-e482-4635-ba65-eb7d42d8e0ea" (UID: "c869bbe6-e482-4635-ba65-eb7d42d8e0ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:16.592213 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.592176 4298 generic.go:356] "Generic (PLEG): container finished" podID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerID="4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac" exitCode=0 Nov 03 23:11:16.592396 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.592235 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" Nov 03 23:11:16.592396 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.592257 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" event={"ID":"c869bbe6-e482-4635-ba65-eb7d42d8e0ea","Type":"ContainerDied","Data":"4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac"} Nov 03 23:11:16.592396 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.592318 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk" event={"ID":"c869bbe6-e482-4635-ba65-eb7d42d8e0ea","Type":"ContainerDied","Data":"67f6614c7127655800d2733decf68d435b5dba6a4e583f8da87ef16630cdff15"} Nov 03 23:11:16.592396 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.592345 4298 scope.go:122] "RemoveContainer" containerID="4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac" Nov 03 23:11:16.601445 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.601424 4298 scope.go:122] "RemoveContainer" containerID="296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e" Nov 03 23:11:16.609586 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.609564 4298 scope.go:122] "RemoveContainer" containerID="4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac" Nov 03 23:11:16.609888 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:11:16.609870 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac\": container with ID starting with 4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac not found: ID does not exist" containerID="4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac" Nov 03 23:11:16.609943 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.609898 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac"} err="failed to get container status \"4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac\": rpc error: code = NotFound desc = could not find container \"4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac\": container with ID starting with 4ec22a1ee7c6095a86c18938c08e3d479e36d8807c05efbeda8e61a62b0de1ac not found: ID does not exist" Nov 03 23:11:16.609943 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.609915 4298 scope.go:122] "RemoveContainer" containerID="296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e" Nov 03 23:11:16.610158 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:11:16.610140 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e\": container with ID starting with 296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e not found: ID does not exist" containerID="296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e" Nov 03 23:11:16.610201 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.610162 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e"} err="failed to get container status \"296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e\": rpc error: code = NotFound desc = could not find container \"296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e\": container with ID starting with 296caecd93f61c32edb76a8a3a65cfb10eaf5522472cb60c23dde420bf1eda0e not found: ID does not exist" Nov 03 23:11:16.644067 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.644033 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpsjg\" (UniqueName: \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-kube-api-access-qpsjg\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:16.644067 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.644065 4298 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-envoy\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:16.644067 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.644075 4298 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-token\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:16.644323 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.644083 4298 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-istio-podinfo\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:16.644323 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.644091 4298 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c869bbe6-e482-4635-ba65-eb7d42d8e0ea-workload-socket\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:16.758306 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.758271 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk"] Nov 03 23:11:16.762060 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:16.762038 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f18d5a31-openshift-default-758bc7bfff-k9xwk"] Nov 03 23:11:17.133722 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:17.133690 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" path="/var/lib/kubelet/pods/c869bbe6-e482-4635-ba65-eb7d42d8e0ea/volumes" Nov 03 23:11:17.552133 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:11:17.552097 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:11:22.485880 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:22.485845 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:22.776197 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:22.776113 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:22.970041 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:22.970010 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:23.156183 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:23.156086 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:23.347337 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:23.347284 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:25.243990 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:25.243957 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:27.273388 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:27.273353 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:27.470117 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:27.470086 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:28.094887 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:28.094851 4298 scope.go:122] "RemoveContainer" containerID="a986f56b79b5c76034855d83f479a4575fe0c41db3ba61ea85ebc86ed82bb5cd" Nov 03 23:11:28.105457 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:28.105433 4298 scope.go:122] "RemoveContainer" containerID="79f85bfa5929bdc2f78b0be10b7b28487196cfc384799c5d4abfd7e6c4427d4d" Nov 03 23:11:29.899489 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:29.899460 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:30.585707 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:30.585666 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:30.758796 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:30.758766 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:31.276975 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.276938 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc"] Nov 03 23:11:31.277376 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.277321 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" Nov 03 23:11:31.277376 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.277332 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" Nov 03 23:11:31.277376 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.277345 4298 cpu_manager.go:414] "containerMap: removing container" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" Nov 03 23:11:31.277376 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.277350 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" Nov 03 23:11:31.277376 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.277367 4298 cpu_manager.go:414] "containerMap: removing container" podUID="3d5646d6-a2c9-4e68-81da-ac2ce456127f" containerName="revoke-keys" Nov 03 23:11:31.277376 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.277372 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5646d6-a2c9-4e68-81da-ac2ce456127f" containerName="revoke-keys" Nov 03 23:11:31.277628 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.277438 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5646d6-a2c9-4e68-81da-ac2ce456127f" containerName="revoke-keys" Nov 03 23:11:31.277628 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.277451 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" Nov 03 23:11:31.277628 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.277457 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="c869bbe6-e482-4635-ba65-eb7d42d8e0ea" containerName="istio-proxy" Nov 03 23:11:31.281878 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.281860 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:11:31.338657 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.338617 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-26c7c9\"" Nov 03 23:11:31.387015 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.386980 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7jzs\" (UniqueName: \"kubernetes.io/projected/b8a41937-6aea-4f73-8022-f225e27edc69-kube-api-access-q7jzs\") pod \"payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc\" (UID: \"b8a41937-6aea-4f73-8022-f225e27edc69\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:11:31.387015 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.387019 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a41937-6aea-4f73-8022-f225e27edc69-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc\" (UID: \"b8a41937-6aea-4f73-8022-f225e27edc69\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:11:31.389367 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.389343 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-26c7c9-dockercfg-qmvmj\"" Nov 03 23:11:31.392469 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.392443 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc"] Nov 03 23:11:31.488577 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.488540 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7jzs\" (UniqueName: \"kubernetes.io/projected/b8a41937-6aea-4f73-8022-f225e27edc69-kube-api-access-q7jzs\") pod \"payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc\" (UID: \"b8a41937-6aea-4f73-8022-f225e27edc69\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:11:31.488577 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.488583 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a41937-6aea-4f73-8022-f225e27edc69-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc\" (UID: \"b8a41937-6aea-4f73-8022-f225e27edc69\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:11:31.489271 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.489250 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a41937-6aea-4f73-8022-f225e27edc69-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc\" (UID: \"b8a41937-6aea-4f73-8022-f225e27edc69\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:11:31.546453 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.546387 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7jzs\" (UniqueName: \"kubernetes.io/projected/b8a41937-6aea-4f73-8022-f225e27edc69-kube-api-access-q7jzs\") pod \"payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc\" (UID: \"b8a41937-6aea-4f73-8022-f225e27edc69\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:11:31.596857 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.596819 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:11:31.961753 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:11:31.961712 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a41937_6aea_4f73_8022_f225e27edc69.slice/crio-d6655a3bd2f4f9e5a4815e15d69e3820e4959d00b77ac919f27d0cb680fbfbe6 WatchSource:0}: Error finding container d6655a3bd2f4f9e5a4815e15d69e3820e4959d00b77ac919f27d0cb680fbfbe6: Status 404 returned error can't find the container with id d6655a3bd2f4f9e5a4815e15d69e3820e4959d00b77ac919f27d0cb680fbfbe6 Nov 03 23:11:31.965979 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:31.965954 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc"] Nov 03 23:11:32.675794 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:32.675760 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" event={"ID":"b8a41937-6aea-4f73-8022-f225e27edc69","Type":"ContainerStarted","Data":"fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803"} Nov 03 23:11:32.676193 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:32.675812 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" event={"ID":"b8a41937-6aea-4f73-8022-f225e27edc69","Type":"ContainerStarted","Data":"d6655a3bd2f4f9e5a4815e15d69e3820e4959d00b77ac919f27d0cb680fbfbe6"} Nov 03 23:11:32.676193 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:32.675880 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:11:32.788954 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:32.788900 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" podStartSLOduration=1.7888850760000001 podStartE2EDuration="1.788885076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:32.788754712 +0000 UTC m=+3849.769488365" watchObservedRunningTime="2026-11-03 23:11:32.788885076 +0000 UTC m=+3849.769618729" Nov 03 23:11:33.857203 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:33.857168 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:34.040975 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:34.040947 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:34.224850 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:34.224818 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:35.543166 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:35.543131 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:38.925388 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:38.925351 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:41.047303 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:41.047265 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7"] Nov 03 23:11:41.097738 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:41.097702 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d04ec62f-bxkn7"] Nov 03 23:11:41.133469 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:41.133436 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="797461df-3131-4e61-a78e-4bccb5b1d854" path="/var/lib/kubelet/pods/797461df-3131-4e61-a78e-4bccb5b1d854/volumes" Nov 03 23:11:43.683568 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:43.683539 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:11:44.131308 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:44.131271 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:47.551566 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:11:47.551526 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:11:49.047372 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:49.047326 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:52.034695 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:52.034630 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:54.319030 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:54.318996 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:59.231476 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:11:59.231443 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:04.335323 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:04.335289 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:09.240180 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:09.240141 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:11.825089 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:11.825052 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:14.250244 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:14.250213 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:17.552711 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:12:17.552674 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:12:19.140954 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:19.140916 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:24.133446 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:24.133415 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:25.570660 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:25.570608 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:25.746548 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:25.746499 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:29.029527 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:29.029494 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:30.697131 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:30.697098 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:34.026444 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:34.026351 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:35.864831 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:35.864800 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:38.942870 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:38.942833 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:43.935906 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:43.935872 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:47.552118 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:12:47.552079 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:12:48.817929 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:48.817894 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:53.844706 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:53.844663 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:58.724883 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:12:58.724841 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:03.727223 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:03.727191 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:06.997894 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:06.997861 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:07.173774 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:07.173736 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:08.633781 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:08.633746 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:12.426393 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.426361 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd"] Nov 03 23:13:12.430058 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.430040 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:12.534056 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.534018 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:13:12.534056 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.534052 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sf592\"" Nov 03 23:13:12.534545 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.534517 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pllf\" (UniqueName: \"kubernetes.io/projected/9c35b437-a5e2-4ab8-afa1-1a625def81cc-kube-api-access-6pllf\") pod \"maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:12.534603 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.534563 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:12.534706 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.534690 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:12.537203 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.537179 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd"] Nov 03 23:13:12.635973 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.635934 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pllf\" (UniqueName: \"kubernetes.io/projected/9c35b437-a5e2-4ab8-afa1-1a625def81cc-kube-api-access-6pllf\") pod \"maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:12.635973 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.635975 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:12.636197 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.635994 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:12.636752 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.636732 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:12.638311 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.638294 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:12.693192 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.693121 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pllf\" (UniqueName: \"kubernetes.io/projected/9c35b437-a5e2-4ab8-afa1-1a625def81cc-kube-api-access-6pllf\") pod \"maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:12.743073 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:12.743032 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:13.107988 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:13:13.107952 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c35b437_a5e2_4ab8_afa1_1a625def81cc.slice/crio-8a17b1f9472ba9d91cc44e003fe56b2f6e9a8209389620e46dc403fb343ef8e8 WatchSource:0}: Error finding container 8a17b1f9472ba9d91cc44e003fe56b2f6e9a8209389620e46dc403fb343ef8e8: Status 404 returned error can't find the container with id 8a17b1f9472ba9d91cc44e003fe56b2f6e9a8209389620e46dc403fb343ef8e8 Nov 03 23:13:13.109975 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:13.109950 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd"] Nov 03 23:13:13.138156 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:13.138103 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" event={"ID":"9c35b437-a5e2-4ab8-afa1-1a625def81cc","Type":"ContainerStarted","Data":"8a17b1f9472ba9d91cc44e003fe56b2f6e9a8209389620e46dc403fb343ef8e8"} Nov 03 23:13:13.731723 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:13.731687 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:13.917338 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:13.917297 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:14.144929 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:14.144891 4298 generic.go:356] "Generic (PLEG): container finished" podID="9c35b437-a5e2-4ab8-afa1-1a625def81cc" containerID="80ef10013d9287cef8d4aad3e2bcb17d0e158718600f395d11553f65be19f5ca" exitCode=0 Nov 03 23:13:14.145124 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:14.144984 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" event={"ID":"9c35b437-a5e2-4ab8-afa1-1a625def81cc","Type":"ContainerDied","Data":"80ef10013d9287cef8d4aad3e2bcb17d0e158718600f395d11553f65be19f5ca"} Nov 03 23:13:15.285757 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:15.285734 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:15.358520 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:15.358485 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-xks-ca\") pod \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " Nov 03 23:13:15.358749 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:15.358568 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c35b437-a5e2-4ab8-afa1-1a625def81cc-kube-api-access-6pllf\" (UniqueName: \"kubernetes.io/projected/9c35b437-a5e2-4ab8-afa1-1a625def81cc-kube-api-access-6pllf\") pod \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " Nov 03 23:13:15.358749 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:15.358595 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-openshift-ca\") pod \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\" (UID: \"9c35b437-a5e2-4ab8-afa1-1a625def81cc\") " Nov 03 23:13:15.359018 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:15.358986 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-openshift-ca" pod "9c35b437-a5e2-4ab8-afa1-1a625def81cc" (UID: "9c35b437-a5e2-4ab8-afa1-1a625def81cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:13:15.360763 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:15.360738 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-xks-ca" pod "9c35b437-a5e2-4ab8-afa1-1a625def81cc" (UID: "9c35b437-a5e2-4ab8-afa1-1a625def81cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:13:15.360853 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:15.360764 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c35b437-a5e2-4ab8-afa1-1a625def81cc-kube-api-access-6pllf" pod "9c35b437-a5e2-4ab8-afa1-1a625def81cc" (UID: "9c35b437-a5e2-4ab8-afa1-1a625def81cc"). InnerVolumeSpecName "kube-api-access-6pllf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:13:15.460208 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:15.460176 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:15.460208 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:15.460210 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pllf\" (UniqueName: \"kubernetes.io/projected/9c35b437-a5e2-4ab8-afa1-1a625def81cc-kube-api-access-6pllf\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:15.460414 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:15.460221 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c35b437-a5e2-4ab8-afa1-1a625def81cc-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:16.155521 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:16.155480 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" event={"ID":"9c35b437-a5e2-4ab8-afa1-1a625def81cc","Type":"ContainerDied","Data":"8a17b1f9472ba9d91cc44e003fe56b2f6e9a8209389620e46dc403fb343ef8e8"} Nov 03 23:13:16.155749 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:16.155530 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a17b1f9472ba9d91cc44e003fe56b2f6e9a8209389620e46dc403fb343ef8e8" Nov 03 23:13:16.155749 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:16.155489 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd" Nov 03 23:13:17.552103 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:13:17.552064 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:13:18.927251 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:18.927218 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:24.110536 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:24.110503 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc"] Nov 03 23:13:24.457110 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:24.457084 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:24.636548 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:24.636512 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:25.947309 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:25.947279 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:28.137824 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:28.137787 4298 scope.go:122] "RemoveContainer" containerID="ede102682bb09d5cce7538bd6ee930ae9f1e1e93a28ea377c79e86a4864e300b" Nov 03 23:13:28.145974 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:28.145954 4298 scope.go:122] "RemoveContainer" containerID="3665f2f8ef58183aa21aede3216f76df2087295e076aad067ff641390593f1a1" Nov 03 23:13:29.111217 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:29.111135 4298 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" podUID="b8a41937-6aea-4f73-8022-f225e27edc69" containerName="payload-pre-processing" containerID="cri-o://fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803" gracePeriod=25 Nov 03 23:13:29.362984 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:29.362912 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:13:29.390984 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:29.390954 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a41937-6aea-4f73-8022-f225e27edc69-kube-api-access-q7jzs\" (UniqueName: \"kubernetes.io/projected/b8a41937-6aea-4f73-8022-f225e27edc69-kube-api-access-q7jzs\") pod \"b8a41937-6aea-4f73-8022-f225e27edc69\" (UID: \"b8a41937-6aea-4f73-8022-f225e27edc69\") " Nov 03 23:13:29.391147 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:29.391001 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a41937-6aea-4f73-8022-f225e27edc69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a41937-6aea-4f73-8022-f225e27edc69-plugins-config-volume\") pod \"b8a41937-6aea-4f73-8022-f225e27edc69\" (UID: \"b8a41937-6aea-4f73-8022-f225e27edc69\") " Nov 03 23:13:29.391394 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:29.391370 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a41937-6aea-4f73-8022-f225e27edc69-plugins-config-volume" pod "b8a41937-6aea-4f73-8022-f225e27edc69" (UID: "b8a41937-6aea-4f73-8022-f225e27edc69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:13:29.393105 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:29.393080 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a41937-6aea-4f73-8022-f225e27edc69-kube-api-access-q7jzs" pod "b8a41937-6aea-4f73-8022-f225e27edc69" (UID: "b8a41937-6aea-4f73-8022-f225e27edc69"). InnerVolumeSpecName "kube-api-access-q7jzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:13:29.491757 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:29.491725 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7jzs\" (UniqueName: \"kubernetes.io/projected/b8a41937-6aea-4f73-8022-f225e27edc69-kube-api-access-q7jzs\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:29.491757 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:29.491755 4298 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a41937-6aea-4f73-8022-f225e27edc69-plugins-config-volume\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:30.220206 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:30.220167 4298 generic.go:356] "Generic (PLEG): container finished" podID="b8a41937-6aea-4f73-8022-f225e27edc69" containerID="fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803" exitCode=0 Nov 03 23:13:30.220400 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:30.220214 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" Nov 03 23:13:30.220400 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:30.220248 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" event={"ID":"b8a41937-6aea-4f73-8022-f225e27edc69","Type":"ContainerDied","Data":"fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803"} Nov 03 23:13:30.220400 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:30.220308 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc" event={"ID":"b8a41937-6aea-4f73-8022-f225e27edc69","Type":"ContainerDied","Data":"d6655a3bd2f4f9e5a4815e15d69e3820e4959d00b77ac919f27d0cb680fbfbe6"} Nov 03 23:13:30.220400 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:30.220332 4298 scope.go:122] "RemoveContainer" containerID="fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803" Nov 03 23:13:30.232550 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:30.232415 4298 scope.go:122] "RemoveContainer" containerID="fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803" Nov 03 23:13:30.232753 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:13:30.232731 4298 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803\": container with ID starting with fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803 not found: ID does not exist" containerID="fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803" Nov 03 23:13:30.232805 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:30.232762 4298 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803"} err="failed to get container status \"fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803\": rpc error: code = NotFound desc = could not find container \"fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803\": container with ID starting with fb2a66feaaed068abeb760b0c6fd6a178181f992096270407310ef65fb0f7803 not found: ID does not exist" Nov 03 23:13:30.385156 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:30.385120 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc"] Nov 03 23:13:30.435544 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:30.435504 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-26c7c9-5f8c8d46b7-jcnnc"] Nov 03 23:13:30.607615 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:30.607533 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:31.133908 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:31.133878 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a41937-6aea-4f73-8022-f225e27edc69" path="/var/lib/kubelet/pods/b8a41937-6aea-4f73-8022-f225e27edc69/volumes" Nov 03 23:13:33.072725 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:33.072694 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:34.618794 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:34.618758 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:34.919988 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:34.919892 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:35.211953 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:35.211922 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd"] Nov 03 23:13:35.261868 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:35.261834 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-26c7c9-kd5jd"] Nov 03 23:13:35.717834 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:35.717803 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:36.219783 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:36.219750 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:37.133622 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:37.133588 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c35b437-a5e2-4ab8-afa1-1a625def81cc" path="/var/lib/kubelet/pods/9c35b437-a5e2-4ab8-afa1-1a625def81cc/volumes" Nov 03 23:13:39.047833 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:39.047798 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld"] Nov 03 23:13:39.097764 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:39.097735 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6mjld"] Nov 03 23:13:39.133776 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:39.133751 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a9f8632-8514-45d6-9061-c77b9715daf8" path="/var/lib/kubelet/pods/8a9f8632-8514-45d6-9061-c77b9715daf8/volumes" Nov 03 23:13:41.337263 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:41.337231 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:41.530842 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:41.530810 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:41.716487 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:13:41.716454 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:47.551295 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:13:47.551250 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:14:03.075688 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:03.073759 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m"] Nov 03 23:14:03.100764 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:03.100723 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6wh7m"] Nov 03 23:14:03.186917 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:03.186876 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2dd391a-8c7a-4145-b57c-96242aff2acb" path="/var/lib/kubelet/pods/f2dd391a-8c7a-4145-b57c-96242aff2acb/volumes" Nov 03 23:14:08.957111 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:08.957077 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n"] Nov 03 23:14:08.957497 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:08.957451 4298 cpu_manager.go:414] "containerMap: removing container" podUID="9c35b437-a5e2-4ab8-afa1-1a625def81cc" containerName="revoke-keys" Nov 03 23:14:08.957497 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:08.957461 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c35b437-a5e2-4ab8-afa1-1a625def81cc" containerName="revoke-keys" Nov 03 23:14:08.957497 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:08.957492 4298 cpu_manager.go:414] "containerMap: removing container" podUID="b8a41937-6aea-4f73-8022-f225e27edc69" containerName="payload-pre-processing" Nov 03 23:14:08.957612 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:08.957498 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a41937-6aea-4f73-8022-f225e27edc69" containerName="payload-pre-processing" Nov 03 23:14:08.957612 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:08.957554 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a41937-6aea-4f73-8022-f225e27edc69" containerName="payload-pre-processing" Nov 03 23:14:08.957612 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:08.957565 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c35b437-a5e2-4ab8-afa1-1a625def81cc" containerName="revoke-keys" Nov 03 23:14:08.960595 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:08.960578 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:09.015301 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.015268 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:14:09.018768 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.018736 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n"] Nov 03 23:14:09.065597 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.065563 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sf592\"" Nov 03 23:14:09.108206 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.108159 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxw4\" (UniqueName: \"kubernetes.io/projected/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-kube-api-access-jkxw4\") pod \"maas-api-revoke-sub-f03629db-8bt9n\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:09.108401 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.108371 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8bt9n\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:09.108474 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.108433 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8bt9n\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:09.209464 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.209376 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8bt9n\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:09.209464 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.209444 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxw4\" (UniqueName: \"kubernetes.io/projected/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-kube-api-access-jkxw4\") pod \"maas-api-revoke-sub-f03629db-8bt9n\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:09.209707 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.209553 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8bt9n\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:09.210230 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.210212 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8bt9n\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:09.211889 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.211868 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8bt9n\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:09.266428 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.266393 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxw4\" (UniqueName: \"kubernetes.io/projected/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-kube-api-access-jkxw4\") pod \"maas-api-revoke-sub-f03629db-8bt9n\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:09.273203 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.273166 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:09.641679 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:14:09.641627 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce94e42_45f1_471f_a6e6_9f5ba47d2ef2.slice/crio-d6d3da23323530f2e4e6d8ed27037dcbab3d48f6bdebf027001a384066c991dd WatchSource:0}: Error finding container d6d3da23323530f2e4e6d8ed27037dcbab3d48f6bdebf027001a384066c991dd: Status 404 returned error can't find the container with id d6d3da23323530f2e4e6d8ed27037dcbab3d48f6bdebf027001a384066c991dd Nov 03 23:14:09.645037 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:09.645014 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n"] Nov 03 23:14:10.427196 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:10.427163 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" event={"ID":"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2","Type":"ContainerStarted","Data":"c963099f71a53ceb3efe5694eb78e64a2971d73cef65dd1676de6430a69d80c8"} Nov 03 23:14:10.427587 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:10.427219 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" event={"ID":"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2","Type":"ContainerStarted","Data":"d6d3da23323530f2e4e6d8ed27037dcbab3d48f6bdebf027001a384066c991dd"} Nov 03 23:14:10.542918 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:10.542871 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" podStartSLOduration=2.542856197 podStartE2EDuration="2.542856197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:14:10.540280392 +0000 UTC m=+4007.521014044" watchObservedRunningTime="2026-11-03 23:14:10.542856197 +0000 UTC m=+4007.523589882" Nov 03 23:14:11.432973 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:11.432936 4298 generic.go:356] "Generic (PLEG): container finished" podID="1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2" containerID="c963099f71a53ceb3efe5694eb78e64a2971d73cef65dd1676de6430a69d80c8" exitCode=0 Nov 03 23:14:11.433365 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:11.433012 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" event={"ID":"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2","Type":"ContainerDied","Data":"c963099f71a53ceb3efe5694eb78e64a2971d73cef65dd1676de6430a69d80c8"} Nov 03 23:14:12.571121 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:12.571097 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:12.741936 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:12.741906 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-openshift-ca\") pod \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " Nov 03 23:14:12.742145 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:12.741950 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-xks-ca\") pod \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " Nov 03 23:14:12.742145 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:12.741975 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-kube-api-access-jkxw4\" (UniqueName: \"kubernetes.io/projected/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-kube-api-access-jkxw4\") pod \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\" (UID: \"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2\") " Nov 03 23:14:12.742341 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:12.742308 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-openshift-ca" pod "1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2" (UID: "1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:14:12.744123 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:12.744098 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-xks-ca" pod "1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2" (UID: "1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:14:12.744569 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:12.744549 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-kube-api-access-jkxw4" pod "1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2" (UID: "1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2"). InnerVolumeSpecName "kube-api-access-jkxw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:14:12.843491 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:12.843446 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:12.843491 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:12.843482 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:12.843491 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:12.843493 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkxw4\" (UniqueName: \"kubernetes.io/projected/1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2-kube-api-access-jkxw4\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:13.443731 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:13.443639 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" Nov 03 23:14:13.443731 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:13.443680 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8bt9n" event={"ID":"1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2","Type":"ContainerDied","Data":"d6d3da23323530f2e4e6d8ed27037dcbab3d48f6bdebf027001a384066c991dd"} Nov 03 23:14:13.443731 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:13.443730 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6d3da23323530f2e4e6d8ed27037dcbab3d48f6bdebf027001a384066c991dd" Nov 03 23:14:17.553872 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:14:17.553832 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:14:26.362493 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:26.362443 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:14:30.982958 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:30.982919 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:14:31.218810 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.218772 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5"] Nov 03 23:14:31.219161 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.219145 4298 cpu_manager.go:414] "containerMap: removing container" podUID="1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2" containerName="revoke-keys" Nov 03 23:14:31.219161 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.219160 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2" containerName="revoke-keys" Nov 03 23:14:31.219309 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.219246 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2" containerName="revoke-keys" Nov 03 23:14:31.222231 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.222209 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:31.280130 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.280051 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5"] Nov 03 23:14:31.323375 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.323339 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg4cp\" (UniqueName: \"kubernetes.io/projected/fe9d7a54-f502-4114-ae83-3ef56d62d16b-kube-api-access-mg4cp\") pod \"maas-api-revoke-sub-0165d7d2-z56b5\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:31.323540 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.323396 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z56b5\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:31.323540 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.323442 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z56b5\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:31.326811 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.326793 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:14:31.327020 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.327002 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sf592\"" Nov 03 23:14:31.424176 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.424142 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z56b5\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:31.424176 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.424177 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z56b5\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:31.424467 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.424265 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg4cp\" (UniqueName: \"kubernetes.io/projected/fe9d7a54-f502-4114-ae83-3ef56d62d16b-kube-api-access-mg4cp\") pod \"maas-api-revoke-sub-0165d7d2-z56b5\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:31.425013 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.424984 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z56b5\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:31.426567 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.426543 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z56b5\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:31.481222 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.481186 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg4cp\" (UniqueName: \"kubernetes.io/projected/fe9d7a54-f502-4114-ae83-3ef56d62d16b-kube-api-access-mg4cp\") pod \"maas-api-revoke-sub-0165d7d2-z56b5\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:31.534785 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.534707 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:31.904766 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:14:31.904724 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9d7a54_f502_4114_ae83_3ef56d62d16b.slice/crio-fa36ce41c73cf0237a25cfdd2c14b5649b0201baa95a8a090bc097a9491c5ffe WatchSource:0}: Error finding container fa36ce41c73cf0237a25cfdd2c14b5649b0201baa95a8a090bc097a9491c5ffe: Status 404 returned error can't find the container with id fa36ce41c73cf0237a25cfdd2c14b5649b0201baa95a8a090bc097a9491c5ffe Nov 03 23:14:31.907173 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:31.907149 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5"] Nov 03 23:14:32.535235 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:32.535148 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" event={"ID":"fe9d7a54-f502-4114-ae83-3ef56d62d16b","Type":"ContainerStarted","Data":"5aae2543db8f38b48686da9c317616a22bc539e28ecc091e657c20036de294ef"} Nov 03 23:14:32.535235 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:32.535201 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" event={"ID":"fe9d7a54-f502-4114-ae83-3ef56d62d16b","Type":"ContainerStarted","Data":"fa36ce41c73cf0237a25cfdd2c14b5649b0201baa95a8a090bc097a9491c5ffe"} Nov 03 23:14:32.653246 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:32.653196 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" podStartSLOduration=1.6531830539999999 podStartE2EDuration="1.653183054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:14:32.650063374 +0000 UTC m=+4029.630797028" watchObservedRunningTime="2026-11-03 23:14:32.653183054 +0000 UTC m=+4029.633916723" Nov 03 23:14:33.047026 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:33.046990 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4"] Nov 03 23:14:33.049421 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:33.049401 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-258fff3b-pk5z4"] Nov 03 23:14:33.139600 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:33.139571 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7008ef7-cdec-4378-b3d9-e6f5eca89431" path="/var/lib/kubelet/pods/e7008ef7-cdec-4378-b3d9-e6f5eca89431/volumes" Nov 03 23:14:33.541046 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:33.541010 4298 generic.go:356] "Generic (PLEG): container finished" podID="fe9d7a54-f502-4114-ae83-3ef56d62d16b" containerID="5aae2543db8f38b48686da9c317616a22bc539e28ecc091e657c20036de294ef" exitCode=0 Nov 03 23:14:33.541467 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:33.541098 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" event={"ID":"fe9d7a54-f502-4114-ae83-3ef56d62d16b","Type":"ContainerDied","Data":"5aae2543db8f38b48686da9c317616a22bc539e28ecc091e657c20036de294ef"} Nov 03 23:14:34.681437 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:34.681414 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:34.755843 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:34.755808 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9d7a54-f502-4114-ae83-3ef56d62d16b-kube-api-access-mg4cp\" (UniqueName: \"kubernetes.io/projected/fe9d7a54-f502-4114-ae83-3ef56d62d16b-kube-api-access-mg4cp\") pod \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " Nov 03 23:14:34.755843 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:34.755846 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-xks-ca\") pod \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " Nov 03 23:14:34.756087 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:34.755931 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-openshift-ca\") pod \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\" (UID: \"fe9d7a54-f502-4114-ae83-3ef56d62d16b\") " Nov 03 23:14:34.756313 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:34.756286 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-openshift-ca" pod "fe9d7a54-f502-4114-ae83-3ef56d62d16b" (UID: "fe9d7a54-f502-4114-ae83-3ef56d62d16b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:14:34.757973 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:34.757952 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-xks-ca" pod "fe9d7a54-f502-4114-ae83-3ef56d62d16b" (UID: "fe9d7a54-f502-4114-ae83-3ef56d62d16b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:14:34.758055 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:34.757956 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9d7a54-f502-4114-ae83-3ef56d62d16b-kube-api-access-mg4cp" pod "fe9d7a54-f502-4114-ae83-3ef56d62d16b" (UID: "fe9d7a54-f502-4114-ae83-3ef56d62d16b"). InnerVolumeSpecName "kube-api-access-mg4cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:14:34.857495 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:34.857415 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:34.857495 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:34.857442 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg4cp\" (UniqueName: \"kubernetes.io/projected/fe9d7a54-f502-4114-ae83-3ef56d62d16b-kube-api-access-mg4cp\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:34.857495 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:34.857452 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9d7a54-f502-4114-ae83-3ef56d62d16b-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:35.550851 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:35.550824 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" Nov 03 23:14:35.550851 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:35.550843 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z56b5" event={"ID":"fe9d7a54-f502-4114-ae83-3ef56d62d16b","Type":"ContainerDied","Data":"fa36ce41c73cf0237a25cfdd2c14b5649b0201baa95a8a090bc097a9491c5ffe"} Nov 03 23:14:35.551077 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:35.550883 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa36ce41c73cf0237a25cfdd2c14b5649b0201baa95a8a090bc097a9491c5ffe" Nov 03 23:14:36.585279 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:36.585250 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:14:42.997740 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:42.997701 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g"] Nov 03 23:14:42.998140 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:42.998076 4298 cpu_manager.go:414] "containerMap: removing container" podUID="fe9d7a54-f502-4114-ae83-3ef56d62d16b" containerName="revoke-keys" Nov 03 23:14:42.998140 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:42.998087 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9d7a54-f502-4114-ae83-3ef56d62d16b" containerName="revoke-keys" Nov 03 23:14:42.998221 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:42.998159 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9d7a54-f502-4114-ae83-3ef56d62d16b" containerName="revoke-keys" Nov 03 23:14:43.001538 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.001518 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:43.060074 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.060039 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g"] Nov 03 23:14:43.105441 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.105413 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:14:43.105627 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.105610 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sf592\"" Nov 03 23:14:43.139564 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.139532 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8k85g\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:43.139564 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.139566 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n277\" (UniqueName: \"kubernetes.io/projected/208e0581-b9a6-4957-aa85-816f0a7910f9-kube-api-access-6n277\") pod \"maas-api-revoke-sub-6b19338f-8k85g\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:43.139774 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.139603 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8k85g\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:43.240853 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.240818 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8k85g\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:43.241079 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.240862 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n277\" (UniqueName: \"kubernetes.io/projected/208e0581-b9a6-4957-aa85-816f0a7910f9-kube-api-access-6n277\") pod \"maas-api-revoke-sub-6b19338f-8k85g\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:43.241079 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.240915 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8k85g\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:43.241535 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.241516 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8k85g\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:43.243229 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.243208 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8k85g\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:43.298270 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.298196 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n277\" (UniqueName: \"kubernetes.io/projected/208e0581-b9a6-4957-aa85-816f0a7910f9-kube-api-access-6n277\") pod \"maas-api-revoke-sub-6b19338f-8k85g\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:43.314068 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.314038 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:43.677681 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:14:43.677629 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208e0581_b9a6_4957_aa85_816f0a7910f9.slice/crio-a6feda94a5b4836c0a3d18df7301fcb8b6ff5125d3a530335676d3057b8b3da5 WatchSource:0}: Error finding container a6feda94a5b4836c0a3d18df7301fcb8b6ff5125d3a530335676d3057b8b3da5: Status 404 returned error can't find the container with id a6feda94a5b4836c0a3d18df7301fcb8b6ff5125d3a530335676d3057b8b3da5 Nov 03 23:14:43.681851 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:43.681828 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g"] Nov 03 23:14:44.591909 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:44.591874 4298 generic.go:356] "Generic (PLEG): container finished" podID="208e0581-b9a6-4957-aa85-816f0a7910f9" containerID="0e4e9a19613b267d512a79ddd98203f8f85e0d2173baca944b3171149670daa1" exitCode=0 Nov 03 23:14:44.592316 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:44.591964 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" event={"ID":"208e0581-b9a6-4957-aa85-816f0a7910f9","Type":"ContainerDied","Data":"0e4e9a19613b267d512a79ddd98203f8f85e0d2173baca944b3171149670daa1"} Nov 03 23:14:44.592316 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:44.592018 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" event={"ID":"208e0581-b9a6-4957-aa85-816f0a7910f9","Type":"ContainerStarted","Data":"a6feda94a5b4836c0a3d18df7301fcb8b6ff5125d3a530335676d3057b8b3da5"} Nov 03 23:14:45.731908 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:45.731885 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:45.866484 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:45.866399 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-xks-ca\") pod \"208e0581-b9a6-4957-aa85-816f0a7910f9\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " Nov 03 23:14:45.866673 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:45.866560 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-openshift-ca\") pod \"208e0581-b9a6-4957-aa85-816f0a7910f9\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " Nov 03 23:14:45.866673 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:45.866595 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208e0581-b9a6-4957-aa85-816f0a7910f9-kube-api-access-6n277\" (UniqueName: \"kubernetes.io/projected/208e0581-b9a6-4957-aa85-816f0a7910f9-kube-api-access-6n277\") pod \"208e0581-b9a6-4957-aa85-816f0a7910f9\" (UID: \"208e0581-b9a6-4957-aa85-816f0a7910f9\") " Nov 03 23:14:45.866982 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:45.866958 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-openshift-ca" pod "208e0581-b9a6-4957-aa85-816f0a7910f9" (UID: "208e0581-b9a6-4957-aa85-816f0a7910f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:14:45.868543 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:45.868516 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-xks-ca" pod "208e0581-b9a6-4957-aa85-816f0a7910f9" (UID: "208e0581-b9a6-4957-aa85-816f0a7910f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:14:45.868693 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:45.868668 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208e0581-b9a6-4957-aa85-816f0a7910f9-kube-api-access-6n277" pod "208e0581-b9a6-4957-aa85-816f0a7910f9" (UID: "208e0581-b9a6-4957-aa85-816f0a7910f9"). InnerVolumeSpecName "kube-api-access-6n277". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:14:45.967534 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:45.967495 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-openshift-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:45.967534 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:45.967527 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n277\" (UniqueName: \"kubernetes.io/projected/208e0581-b9a6-4957-aa85-816f0a7910f9-kube-api-access-6n277\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:45.967771 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:45.967555 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208e0581-b9a6-4957-aa85-816f0a7910f9-maas-api-xks-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:46.602679 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:46.602627 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" event={"ID":"208e0581-b9a6-4957-aa85-816f0a7910f9","Type":"ContainerDied","Data":"a6feda94a5b4836c0a3d18df7301fcb8b6ff5125d3a530335676d3057b8b3da5"} Nov 03 23:14:46.602870 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:46.602700 4298 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6feda94a5b4836c0a3d18df7301fcb8b6ff5125d3a530335676d3057b8b3da5" Nov 03 23:14:46.602870 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:14:46.602633 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8k85g" Nov 03 23:14:47.551068 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:14:47.551034 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:15:00.160806 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.160771 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6"] Nov 03 23:15:00.161310 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.161171 4298 cpu_manager.go:414] "containerMap: removing container" podUID="208e0581-b9a6-4957-aa85-816f0a7910f9" containerName="revoke-keys" Nov 03 23:15:00.161310 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.161183 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208e0581-b9a6-4957-aa85-816f0a7910f9" containerName="revoke-keys" Nov 03 23:15:00.161310 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.161270 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="208e0581-b9a6-4957-aa85-816f0a7910f9" containerName="revoke-keys" Nov 03 23:15:00.168261 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.168241 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" Nov 03 23:15:00.222603 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.222566 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a9b466-ed66-4fea-aa54-54b1e35be293-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-nkmt6\" (UID: \"a4a9b466-ed66-4fea-aa54-54b1e35be293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" Nov 03 23:15:00.222819 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.222675 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjs24\" (UniqueName: \"kubernetes.io/projected/a4a9b466-ed66-4fea-aa54-54b1e35be293-kube-api-access-wjs24\") pod \"maas-api-key-cleanup-29895795-nkmt6\" (UID: \"a4a9b466-ed66-4fea-aa54-54b1e35be293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" Nov 03 23:15:00.224156 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.224133 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6"] Nov 03 23:15:00.271059 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.271023 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sf592\"" Nov 03 23:15:00.323897 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.323866 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjs24\" (UniqueName: \"kubernetes.io/projected/a4a9b466-ed66-4fea-aa54-54b1e35be293-kube-api-access-wjs24\") pod \"maas-api-key-cleanup-29895795-nkmt6\" (UID: \"a4a9b466-ed66-4fea-aa54-54b1e35be293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" Nov 03 23:15:00.324067 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.323957 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a9b466-ed66-4fea-aa54-54b1e35be293-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-nkmt6\" (UID: \"a4a9b466-ed66-4fea-aa54-54b1e35be293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" Nov 03 23:15:00.324540 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.324521 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a9b466-ed66-4fea-aa54-54b1e35be293-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-nkmt6\" (UID: \"a4a9b466-ed66-4fea-aa54-54b1e35be293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" Nov 03 23:15:00.381040 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.381012 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjs24\" (UniqueName: \"kubernetes.io/projected/a4a9b466-ed66-4fea-aa54-54b1e35be293-kube-api-access-wjs24\") pod \"maas-api-key-cleanup-29895795-nkmt6\" (UID: \"a4a9b466-ed66-4fea-aa54-54b1e35be293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" Nov 03 23:15:00.480280 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.480250 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" Nov 03 23:15:00.849254 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:15:00.849210 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a9b466_ed66_4fea_aa54_54b1e35be293.slice/crio-25f5752eb6cf7b88570df58fc1d8e915609438fd5a2909b41161832d4fa63b3b WatchSource:0}: Error finding container 25f5752eb6cf7b88570df58fc1d8e915609438fd5a2909b41161832d4fa63b3b: Status 404 returned error can't find the container with id 25f5752eb6cf7b88570df58fc1d8e915609438fd5a2909b41161832d4fa63b3b Nov 03 23:15:00.850611 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:00.850587 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6"] Nov 03 23:15:01.698767 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:01.698735 4298 generic.go:356] "Generic (PLEG): container finished" podID="a4a9b466-ed66-4fea-aa54-54b1e35be293" containerID="f97f39bcb417679b01f7f708f12f85125f04f23c19281f7c337d88521db0ddae" exitCode=60 Nov 03 23:15:01.699174 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:01.698816 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" event={"ID":"a4a9b466-ed66-4fea-aa54-54b1e35be293","Type":"ContainerDied","Data":"f97f39bcb417679b01f7f708f12f85125f04f23c19281f7c337d88521db0ddae"} Nov 03 23:15:01.699174 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:01.698867 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" event={"ID":"a4a9b466-ed66-4fea-aa54-54b1e35be293","Type":"ContainerStarted","Data":"25f5752eb6cf7b88570df58fc1d8e915609438fd5a2909b41161832d4fa63b3b"} Nov 03 23:15:01.699174 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:01.699036 4298 scope.go:122] "RemoveContainer" containerID="f97f39bcb417679b01f7f708f12f85125f04f23c19281f7c337d88521db0ddae" Nov 03 23:15:02.705681 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:02.705630 4298 generic.go:356] "Generic (PLEG): container finished" podID="a4a9b466-ed66-4fea-aa54-54b1e35be293" containerID="1ad066160bdf19404df8aeba80965d5d63ae4bbf2201c7148815dc366f8a0cd8" exitCode=60 Nov 03 23:15:02.706217 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:02.705717 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" event={"ID":"a4a9b466-ed66-4fea-aa54-54b1e35be293","Type":"ContainerDied","Data":"1ad066160bdf19404df8aeba80965d5d63ae4bbf2201c7148815dc366f8a0cd8"} Nov 03 23:15:02.706217 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:02.705787 4298 scope.go:122] "RemoveContainer" containerID="f97f39bcb417679b01f7f708f12f85125f04f23c19281f7c337d88521db0ddae" Nov 03 23:15:02.706217 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:02.706053 4298 scope.go:122] "RemoveContainer" containerID="1ad066160bdf19404df8aeba80965d5d63ae4bbf2201c7148815dc366f8a0cd8" Nov 03 23:15:02.706373 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:15:02.706326 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-nkmt6_odh-ai-gateway-infra(a4a9b466-ed66-4fea-aa54-54b1e35be293)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" podUID="a4a9b466-ed66-4fea-aa54-54b1e35be293" Nov 03 23:15:03.712236 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:03.712207 4298 scope.go:122] "RemoveContainer" containerID="1ad066160bdf19404df8aeba80965d5d63ae4bbf2201c7148815dc366f8a0cd8" Nov 03 23:15:03.712691 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:15:03.712476 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-nkmt6_odh-ai-gateway-infra(a4a9b466-ed66-4fea-aa54-54b1e35be293)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" podUID="a4a9b466-ed66-4fea-aa54-54b1e35be293" Nov 03 23:15:05.227025 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:05.226977 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:15:06.509321 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:06.509292 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ltvt5_845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977/manager/0.log" Nov 03 23:15:07.088713 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:07.088681 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hnz7l_d2962c9c-7179-4c8b-8bc0-bae9e785b340/manager/0.log" Nov 03 23:15:07.969783 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:07.969752 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xhcp4_6d9e9bcd-c138-4660-b91f-d76fc7c48e65/server/0.log" Nov 03 23:15:08.456310 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:08.456279 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-v4vlc_08b579ef-92ab-4fa5-a153-6be536d77795/proxy/0.log" Nov 03 23:15:09.817479 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:09.817453 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29895795-nkmt6_a4a9b466-ed66-4fea-aa54-54b1e35be293/cleanup/1.log" Nov 03 23:15:10.057518 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:10.057488 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-z56b5_fe9d7a54-f502-4114-ae83-3ef56d62d16b/revoke-keys/0.log" Nov 03 23:15:10.297481 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:10.297446 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-8k85g_208e0581-b9a6-4957-aa85-816f0a7910f9/revoke-keys/0.log" Nov 03 23:15:10.535468 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:10.535443 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8bt9n_1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2/revoke-keys/0.log" Nov 03 23:15:10.756255 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:10.756224 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-s66dj_8bd983da-1d6f-4af1-a83d-3b2262efbd62/postgres/0.log" Nov 03 23:15:11.240055 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:11.240028 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29895795-nkmt6_a4a9b466-ed66-4fea-aa54-54b1e35be293/cleanup/1.log" Nov 03 23:15:13.473552 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:13.473526 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-75lm5_30487de4-4746-4395-befd-7cf626ee1bf8/kuadrant-console-plugin/0.log" Nov 03 23:15:14.285061 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:14.285030 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fg9bl_26112fa6-98ae-46b6-b961-292291a69e55/manager/0.log" Nov 03 23:15:15.128169 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:15.128049 4298 scope.go:122] "RemoveContainer" containerID="1ad066160bdf19404df8aeba80965d5d63ae4bbf2201c7148815dc366f8a0cd8" Nov 03 23:15:15.182081 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:15.182050 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-597d745469jn4qn_b76c20eb-1d4b-408a-a86f-38aa1198c340/istio-proxy/0.log" Nov 03 23:15:15.770930 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:15.770892 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" event={"ID":"a4a9b466-ed66-4fea-aa54-54b1e35be293","Type":"ContainerStarted","Data":"24e17e45ec03c447ec33519f49499f582b02e3999e5436c192d83fb05624bf69"} Nov 03 23:15:15.890592 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:15.890539 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" podStartSLOduration=15.890525274 podStartE2EDuration="15.890525274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:15:15.888360273 +0000 UTC m=+4072.869093917" watchObservedRunningTime="2026-11-03 23:15:15.890525274 +0000 UTC m=+4072.871258923" Nov 03 23:15:16.607754 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:16.607724 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-6rjg7_da4f3451-f4d3-408e-aad0-67f89b6f7260/istio-proxy/0.log" Nov 03 23:15:16.777704 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:16.777672 4298 generic.go:356] "Generic (PLEG): container finished" podID="a4a9b466-ed66-4fea-aa54-54b1e35be293" containerID="24e17e45ec03c447ec33519f49499f582b02e3999e5436c192d83fb05624bf69" exitCode=60 Nov 03 23:15:16.777886 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:16.777744 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" event={"ID":"a4a9b466-ed66-4fea-aa54-54b1e35be293","Type":"ContainerDied","Data":"24e17e45ec03c447ec33519f49499f582b02e3999e5436c192d83fb05624bf69"} Nov 03 23:15:16.777886 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:16.777806 4298 scope.go:122] "RemoveContainer" containerID="1ad066160bdf19404df8aeba80965d5d63ae4bbf2201c7148815dc366f8a0cd8" Nov 03 23:15:16.778145 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:16.778128 4298 scope.go:122] "RemoveContainer" containerID="24e17e45ec03c447ec33519f49499f582b02e3999e5436c192d83fb05624bf69" Nov 03 23:15:16.778397 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:15:16.778377 4298 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-nkmt6_odh-ai-gateway-infra(a4a9b466-ed66-4fea-aa54-54b1e35be293)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" podUID="a4a9b466-ed66-4fea-aa54-54b1e35be293" Nov 03 23:15:16.899029 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:16.898948 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6"] Nov 03 23:15:16.899833 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:16.899812 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-6x4mq_818a85a1-2bd0-4d9e-84ef-951f647247e2/istio-proxy/0.log" Nov 03 23:15:17.552536 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:15:17.552497 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:15:17.916717 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:17.916693 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" Nov 03 23:15:17.952028 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:17.952000 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-tqdf6_701f8c70-a62b-487e-8008-04258f4d75b2/istio-proxy/0.log" Nov 03 23:15:17.991363 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:17.991329 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a9b466-ed66-4fea-aa54-54b1e35be293-kube-api-access-wjs24\" (UniqueName: \"kubernetes.io/projected/a4a9b466-ed66-4fea-aa54-54b1e35be293-kube-api-access-wjs24\") pod \"a4a9b466-ed66-4fea-aa54-54b1e35be293\" (UID: \"a4a9b466-ed66-4fea-aa54-54b1e35be293\") " Nov 03 23:15:17.991540 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:17.991375 4298 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a9b466-ed66-4fea-aa54-54b1e35be293-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a9b466-ed66-4fea-aa54-54b1e35be293-maas-api-service-ca\") pod \"a4a9b466-ed66-4fea-aa54-54b1e35be293\" (UID: \"a4a9b466-ed66-4fea-aa54-54b1e35be293\") " Nov 03 23:15:17.991829 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:17.991805 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a9b466-ed66-4fea-aa54-54b1e35be293-maas-api-service-ca" pod "a4a9b466-ed66-4fea-aa54-54b1e35be293" (UID: "a4a9b466-ed66-4fea-aa54-54b1e35be293"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:17.993435 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:17.993407 4298 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a9b466-ed66-4fea-aa54-54b1e35be293-kube-api-access-wjs24" pod "a4a9b466-ed66-4fea-aa54-54b1e35be293" (UID: "a4a9b466-ed66-4fea-aa54-54b1e35be293"). InnerVolumeSpecName "kube-api-access-wjs24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:18.092729 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:18.092616 4298 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjs24\" (UniqueName: \"kubernetes.io/projected/a4a9b466-ed66-4fea-aa54-54b1e35be293-kube-api-access-wjs24\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:18.092729 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:18.092675 4298 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a9b466-ed66-4fea-aa54-54b1e35be293-maas-api-service-ca\") on node \"ip-10-0-139-254.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:18.721779 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:18.721748 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s_d1eeca1b-876a-431b-b98c-eea04180c345/istio-proxy/0.log" Nov 03 23:15:18.790532 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:18.790493 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" event={"ID":"a4a9b466-ed66-4fea-aa54-54b1e35be293","Type":"ContainerDied","Data":"25f5752eb6cf7b88570df58fc1d8e915609438fd5a2909b41161832d4fa63b3b"} Nov 03 23:15:18.790532 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:18.790515 4298 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6" Nov 03 23:15:18.790830 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:18.790556 4298 scope.go:122] "RemoveContainer" containerID="24e17e45ec03c447ec33519f49499f582b02e3999e5436c192d83fb05624bf69" Nov 03 23:15:18.956678 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:18.956634 4298 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6"] Nov 03 23:15:18.960316 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:18.960291 4298 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-nkmt6"] Nov 03 23:15:19.133777 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:19.133705 4298 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a9b466-ed66-4fea-aa54-54b1e35be293" path="/var/lib/kubelet/pods/a4a9b466-ed66-4fea-aa54-54b1e35be293/volumes" Nov 03 23:15:19.235091 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:19.235062 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7b8kb_416fda64-c90d-4b05-b7e2-4f963d168b50/payload-processing/0.log" Nov 03 23:15:26.659280 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:26.659243 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:15:28.209562 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:28.209528 4298 scope.go:122] "RemoveContainer" containerID="93c8cc6dff0211b7797e62f57c0cb28e30ebabe70510251cfd00c50ee7bca986" Nov 03 23:15:28.217750 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:28.217727 4298 scope.go:122] "RemoveContainer" containerID="575398c59534c242d3159692b90e81042f5a0d59e611b35978931809cf431289" Nov 03 23:15:31.319696 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:31.319637 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:15:32.265987 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:32.265955 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ltvt5_845f0e2e-e3d7-4ac7-ad6e-7c80b50f0977/manager/0.log" Nov 03 23:15:32.867908 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:32.867880 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hnz7l_d2962c9c-7179-4c8b-8bc0-bae9e785b340/manager/0.log" Nov 03 23:15:33.638095 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:33.638066 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xhcp4_6d9e9bcd-c138-4660-b91f-d76fc7c48e65/server/0.log" Nov 03 23:15:34.124071 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:34.124044 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-v4vlc_08b579ef-92ab-4fa5-a153-6be536d77795/proxy/0.log" Nov 03 23:15:35.234886 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:35.234853 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-z56b5_fe9d7a54-f502-4114-ae83-3ef56d62d16b/revoke-keys/0.log" Nov 03 23:15:35.462816 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:35.462788 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-8k85g_208e0581-b9a6-4957-aa85-816f0a7910f9/revoke-keys/0.log" Nov 03 23:15:35.682693 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:35.682565 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8bt9n_1ce94e42-45f1-471f-a6e6-9f5ba47d2ef2/revoke-keys/0.log" Nov 03 23:15:35.913311 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:35.913281 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-s66dj_8bd983da-1d6f-4af1-a83d-3b2262efbd62/postgres/0.log" Nov 03 23:15:36.939770 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:36.939734 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:15:38.017309 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:38.017280 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-75lm5_30487de4-4746-4395-befd-7cf626ee1bf8/kuadrant-console-plugin/0.log" Nov 03 23:15:38.746448 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:38.746419 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fg9bl_26112fa6-98ae-46b6-b961-292291a69e55/manager/0.log" Nov 03 23:15:39.492346 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:39.492302 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-597d745469jn4qn_b76c20eb-1d4b-408a-a86f-38aa1198c340/istio-proxy/0.log" Nov 03 23:15:40.933487 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:40.933454 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-6rjg7_da4f3451-f4d3-408e-aad0-67f89b6f7260/istio-proxy/0.log" Nov 03 23:15:41.224315 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:41.224281 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-6x4mq_818a85a1-2bd0-4d9e-84ef-951f647247e2/istio-proxy/0.log" Nov 03 23:15:42.259044 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:42.259013 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-tqdf6_701f8c70-a62b-487e-8008-04258f4d75b2/istio-proxy/0.log" Nov 03 23:15:43.037537 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:43.037502 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-85f45cdb7-dbr2s_d1eeca1b-876a-431b-b98c-eea04180c345/istio-proxy/0.log" Nov 03 23:15:43.548059 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:15:43.548031 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7b8kb_416fda64-c90d-4b05-b7e2-4f963d168b50/payload-processing/0.log" Nov 03 23:15:47.552543 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:15:47.552505 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:16:15.045499 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:16:15.045467 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7b8kb_416fda64-c90d-4b05-b7e2-4f963d168b50/payload-processing/0.log" Nov 03 23:16:15.278958 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:16:15.278930 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7b8kb_416fda64-c90d-4b05-b7e2-4f963d168b50/payload-processing/0.log" Nov 03 23:16:17.551995 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:16:17.551960 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:16:27.049749 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:16:27.049713 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:27.525085 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:16:27.525040 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:28.245380 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:16:28.245349 4298 scope.go:122] "RemoveContainer" containerID="9a5cda43c535574ec754027cf599f5ed8db67f01a8d5ed249011b0328452c62a" Nov 03 23:16:31.668067 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:16:31.668023 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:37.283900 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:16:37.283866 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:47.551729 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:16:47.551691 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:16:52.954074 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:16:52.954045 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7b8kb_416fda64-c90d-4b05-b7e2-4f963d168b50/payload-processing/0.log" Nov 03 23:16:53.182538 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:16:53.182510 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7b8kb_416fda64-c90d-4b05-b7e2-4f963d168b50/payload-processing/0.log" Nov 03 23:16:53.585150 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:16:53.585116 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:53.884282 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:16:53.884204 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:02.705904 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:02.705870 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7zmnf_a0475429-6780-4dd9-8185-199a83ce41f5/global-pull-secret-syncer/0.log" Nov 03 23:17:05.992300 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:05.992266 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" Nov 03 23:17:07.894950 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:07.894916 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-254.us-west-2.compute.internal_815f1999fc2bcba4fa27ffca21498e4f/haproxy/0.log" Nov 03 23:17:13.525417 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.525379 4298 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7"] Nov 03 23:17:13.525871 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.525842 4298 cpu_manager.go:414] "containerMap: removing container" podUID="a4a9b466-ed66-4fea-aa54-54b1e35be293" containerName="cleanup" Nov 03 23:17:13.525871 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.525857 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a9b466-ed66-4fea-aa54-54b1e35be293" containerName="cleanup" Nov 03 23:17:13.525987 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.525899 4298 cpu_manager.go:414] "containerMap: removing container" podUID="a4a9b466-ed66-4fea-aa54-54b1e35be293" containerName="cleanup" Nov 03 23:17:13.525987 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.525905 4298 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a9b466-ed66-4fea-aa54-54b1e35be293" containerName="cleanup" Nov 03 23:17:13.525987 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.525970 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a9b466-ed66-4fea-aa54-54b1e35be293" containerName="cleanup" Nov 03 23:17:13.525987 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.525979 4298 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a9b466-ed66-4fea-aa54-54b1e35be293" containerName="cleanup" Nov 03 23:17:13.528927 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.528910 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.581859 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.581822 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9px9d\"/\"kube-root-ca.crt\"" Nov 03 23:17:13.582009 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.581868 4298 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9px9d\"/\"openshift-service-ca.crt\"" Nov 03 23:17:13.590798 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.585805 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-sys\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.590798 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.585882 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-proc\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.590798 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.585926 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-podres\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.590798 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.585986 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-lib-modules\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.590798 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.586055 4298 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kpq7\" (UniqueName: \"kubernetes.io/projected/139742df-7c49-41f7-b3c3-affb54cdacb9-kube-api-access-8kpq7\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.632164 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.632133 4298 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9px9d\"/\"default-dockercfg-546s9\"" Nov 03 23:17:13.635035 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.634996 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7"] Nov 03 23:17:13.687326 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.687298 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kpq7\" (UniqueName: \"kubernetes.io/projected/139742df-7c49-41f7-b3c3-affb54cdacb9-kube-api-access-8kpq7\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.687485 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.687353 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-sys\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.687485 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.687387 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-proc\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.687485 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.687411 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-podres\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.687485 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.687441 4298 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-lib-modules\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.687664 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.687507 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-proc\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.687664 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.687507 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-sys\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.687664 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.687549 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-lib-modules\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.687664 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.687549 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/139742df-7c49-41f7-b3c3-affb54cdacb9-podres\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.744248 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.744218 4298 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kpq7\" (UniqueName: \"kubernetes.io/projected/139742df-7c49-41f7-b3c3-affb54cdacb9-kube-api-access-8kpq7\") pod \"perf-node-gather-daemonset-j2qq7\" (UID: \"139742df-7c49-41f7-b3c3-affb54cdacb9\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.841150 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.841072 4298 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:13.857587 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:13.857560 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-75lm5_30487de4-4746-4395-befd-7cf626ee1bf8/kuadrant-console-plugin/0.log" Nov 03 23:17:14.204463 ip-10-0-139-254 kubenswrapper[4298]: W1103 23:17:14.204429 4298 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod139742df_7c49_41f7_b3c3_affb54cdacb9.slice/crio-ddb0c7207c6cb593708af20c6d05d09c8a82fa2331be298dd2831e12b3221ffb WatchSource:0}: Error finding container ddb0c7207c6cb593708af20c6d05d09c8a82fa2331be298dd2831e12b3221ffb: Status 404 returned error can't find the container with id ddb0c7207c6cb593708af20c6d05d09c8a82fa2331be298dd2831e12b3221ffb Nov 03 23:17:14.207278 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:14.207242 4298 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7"] Nov 03 23:17:14.329947 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:14.329916 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" event={"ID":"139742df-7c49-41f7-b3c3-affb54cdacb9","Type":"ContainerStarted","Data":"c41274a7f55aabf2280afc3140f3718062a1df0ee46d288961618d7e39c5f0a0"} Nov 03 23:17:14.330116 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:14.329963 4298 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" event={"ID":"139742df-7c49-41f7-b3c3-affb54cdacb9","Type":"ContainerStarted","Data":"ddb0c7207c6cb593708af20c6d05d09c8a82fa2331be298dd2831e12b3221ffb"} Nov 03 23:17:14.330116 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:14.330053 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:14.447452 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:14.447402 4298 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" podStartSLOduration=1.447389483 podStartE2EDuration="1.447389483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:14.446804444 +0000 UTC m=+4191.427538096" watchObservedRunningTime="2026-11-03 23:17:14.447389483 +0000 UTC m=+4191.428123137" Nov 03 23:17:15.155957 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:15.155925 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fg9bl_26112fa6-98ae-46b6-b961-292291a69e55/manager/0.log" Nov 03 23:17:17.550924 ip-10-0-139-254 kubenswrapper[4298]: E1103 23:17:17.550887 4298 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56cpk" containerName="extractor" Nov 03 23:17:17.882370 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:17.882291 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_058b48e3-5571-4d76-a4a0-efb0eced09ee/alertmanager/0.log" Nov 03 23:17:18.307634 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:18.307604 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_058b48e3-5571-4d76-a4a0-efb0eced09ee/config-reloader/0.log" Nov 03 23:17:18.732147 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:18.732120 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_058b48e3-5571-4d76-a4a0-efb0eced09ee/kube-rbac-proxy-web/0.log" Nov 03 23:17:19.156882 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:19.156799 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_058b48e3-5571-4d76-a4a0-efb0eced09ee/kube-rbac-proxy/0.log" Nov 03 23:17:19.587306 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:19.587275 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_058b48e3-5571-4d76-a4a0-efb0eced09ee/kube-rbac-proxy-metric/0.log" Nov 03 23:17:20.011128 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:20.011098 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_058b48e3-5571-4d76-a4a0-efb0eced09ee/prom-label-proxy/0.log" Nov 03 23:17:20.344081 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:20.344005 4298 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-j2qq7" Nov 03 23:17:20.435900 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:20.435874 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_058b48e3-5571-4d76-a4a0-efb0eced09ee/init-config-reloader/0.log" Nov 03 23:17:23.061613 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:23.061582 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-lwmf7_f6231d72-95c6-44f1-adde-246081a3665c/monitoring-plugin/0.log" Nov 03 23:17:23.492338 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:23.492315 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m54ln_8cd23ca2-0297-4e50-b026-3208a1244520/node-exporter/0.log" Nov 03 23:17:23.917082 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:23.917011 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m54ln_8cd23ca2-0297-4e50-b026-3208a1244520/kube-rbac-proxy/0.log" Nov 03 23:17:24.342984 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:24.342959 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m54ln_8cd23ca2-0297-4e50-b026-3208a1244520/init-textfile/0.log" Nov 03 23:17:27.361864 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:27.361828 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:28.263741 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:28.263707 4298 scope.go:122] "RemoveContainer" containerID="c147f63c83b77b1b660c035356996ca45e55f0f201f8c0747909615e239a5c32" Nov 03 23:17:28.519747 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:28.519662 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70e4fb16-2a0d-48f5-b326-f7ce057042f0/prometheus/0.log" Nov 03 23:17:29.046779 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:29.046752 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70e4fb16-2a0d-48f5-b326-f7ce057042f0/config-reloader/0.log" Nov 03 23:17:29.475067 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:29.475039 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70e4fb16-2a0d-48f5-b326-f7ce057042f0/thanos-sidecar/0.log" Nov 03 23:17:29.899716 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:29.899627 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70e4fb16-2a0d-48f5-b326-f7ce057042f0/kube-rbac-proxy-web/0.log" Nov 03 23:17:30.325375 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:30.325346 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70e4fb16-2a0d-48f5-b326-f7ce057042f0/kube-rbac-proxy/0.log" Nov 03 23:17:30.750737 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:30.750710 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70e4fb16-2a0d-48f5-b326-f7ce057042f0/kube-rbac-proxy-thanos/0.log" Nov 03 23:17:31.177386 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:31.177298 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70e4fb16-2a0d-48f5-b326-f7ce057042f0/init-config-reloader/0.log" Nov 03 23:17:32.011914 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:32.011877 4298 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b94tl_cc56646d-7901-4a14-a4af-46d2582ccc5c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:36.943482 ip-10-0-139-254 kubenswrapper[4298]: I1103 23:17:36.943453 4298 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-597d745469jn4qn_b76c20eb-1d4b-408a-a86f-38aa1198c340/istio-proxy/0.log"